Loading ...
Global Do...
News & Politics
9
0
Try Now
Log In
Pricing
DASH page 1 DASH page 2 Multiloader Light-weight Python Object <-> XML Exchange DASH page 3 Multiloader Design goals: • Transparent layer between XML and Python objects. • Allow access to the full XML object model. - Attributes as well as elements • Rapid loading and saving. • Modularity and reusability. - Independence from the main body of code. - Separately tested and documented. DASH page 4 Multiloader – Basic idea • Write out Python objects to XML files. • Read in Python objects from XML files. • Determine the XML structure of the objects using callback functions. • Properly represent object hierarchies (nesting) when saving to and loading from XML. DASH page 5 Multiloader – Example import multiloader2 class TestObject: def XMLName(self): return “TestObject” def XMLFields(self): return {“Version”:’vers’, “Time”:’time’} def XMLAttributes(self): return {“TestObject”:[(“ID”, ‘id’)], “Version”:[(“Revision”,’rev’)]} def XMLEmbedding(self): return {} class_list = [TestObject] m = multiloader2.Multiloader() m.load("database.xml", class_list) print “Loading...” print m.getObjects() print "Loaded...“ print “Saving...” m.save("database2.xml", m.getObjects()) print "Saved..." database.xml <Multiloader> <TestObject ID=“325”> <Version Revision=“12”>5.543a</Version> <Time>8127391</Time> </TestObject> <TestObject> <Time>1681890</Time> </TestObject> </Multiloader> DASH page 6 Multiloader – Callbacks <TestObject ID=“325”> <Version Revision=“12”>5.543a</Version> <Time>8127391</Time> </TestObject> • Four callback functions define the XML structure of an object. - obj.XMLName() returns the XML name of the object. “TestObject” - obj.XMLFields() maps the data members of a Python object to their associated XML element names. {“Version” : “vers”, “Time” : “time”} - obj.XMLAttributes() maps XML element names into tuples, where each tuple consists of an attribute for that element to expect, and the Python object’s data member that stores the attribute. {“TestObject” : [(“ID”, “id”)], “Version” : [(“Revision”, “rev”)] } - obj.XMLEmbedding() maps XML object names to the constructors, for objects that are nested within the object. DASH page 7 Multiloader – Usage (Loading) Once you have an object with the proper callbacks defined, you can use Multiloader. class_list = [TestObject] m = multiloader2.Multiloader() m.load("database.xml", class_list) • The load function takes a filename and a list of classes. • It will parse through the XML file and create Python objects out of all of the second-level XML instances of those objects that it can find. - Multiloader expects that at the top level, there is a single XML object. • It will also reconstruct the nesting hierarchy through use of the data in the XMLEmbedding callback. • The loadString(string, class_list) method can be used to have multiloader parse a file stored in the form of a string, rather than actually reading in a file. DASH page 8 Multiloader – Usage (Saving) • Accessing the data and saving are extremely simple. m.save("database2.xml", m.getObjects()) • The getObjects() function returns a dictionary of lists of objects, keyed by the XML names of the objects. • The save(filename, data) function accepts a dictionary of this form and saves it to the given file. • The saveList(filename, list) function accepts a regular list of objects and saves them in the same fashion. DASH page 9 WidgetPy Python Web Programming Made Simple(r) DASH page 10 WidgetPy Design goals: • Minimize HTML coding for forms • Adhere to Python object model • Utilize CSS for presentation whenever possible • Enhanced user experience - Javascript form validation • Provide extensibility and flexibility • 80/20 rule – this is an 80% solution DASH page 11 WidgetPy – Basic idea Link HTML “widgets” to Python objects Widgets should “know” how to display themselves Provide mechanisms to reinstantiate objects upon form submission Use keyword arguments for options DASH page 12 WidgetPy – Example #! /usr/local/bin/python import cgi import cgitb; cgitb.enable() import widget # (very) simple object class testobj: id = 0 def __init__(self): self.id = testobj.id testobj.id += 1 self.data = None # Mainline print "Content-type: text/html\n" # Get the form data (if there is any) inForm = cgi.FieldStorage() objDict = widget.formProc(inForm, {"testobj":testobj}) # formProc returns a dictionary of the form: {"obj<objID>":object,..} if objDict: for k in objDict.keys(): obj = objDict[k] else: obj = testobj() obj.data = "Testing" try: # Simplest form: no caption, no example tw = widget.pwText(obj, "data") # Put the text widget in a form form = widget.pwForm([tw]) # Wrap the form in a header page = widget.pwHeader([form], title="Test Page") except widget.NoSuchAttribute: print widget.NoSuchAttribute print page.display(0) DASH page 13 WidgetPy – Example DASH page 14 WidgetPy – Details Objects must have an empty constructor • If you plan on using formProc You can add your own widgets • Must support a “display” attribute • Takes one argument (level) - Indicates the number of indents Default presentation model is 3 column: • Caption (optional) • Value • Example (optional) DASH page 15 WidgetPy – Components Four major components: • Main program (provided by user) • WidgetPy package (generates HTML) • CSS files (to handle positioning and presentation) - Stored in “../css/” - Default css files provided - Extension mechanism available for custom css • Javascript files - Stored in “../js/” - widget.js: Support for the widget library (submission, validation, etc.) - domtabs.js: Support for tabs - showstyle.js: Support for switchable, alternate stylesheets • Image files - Stored in “../images/” DASH page 16 Non-Form Widgets pwSimple pwParagraph pwCaption pwLink pwList pwListItem pwError pwHeader DASH page 17 WidgetPy – pwSimple pwSimple – output text Arguments: • None Keywords: • divClass – override the default class of “pwSimple” • id – the element ID of the field • text – text to output within an HTML <div> • textOnly – text to output (no surrounding HTML) DASH page 18 WidgetPy – pwParagraph pwParagraph – display a paragraph Arguments: • text – the text of the paragraph itself Keywords: • caption – the field caption text (e.g. “Name”) • help – help link (e.g. “return help();”) • divClass – override the default class of “pwParagraph” • id – the element ID of the field • fieldSpecial – special options for the input field DASH page 19 WidgetPy - pwCaption pwCaption – display a caption (mostly for internal use) Arguments: • None Keywords: • caption – the field caption text (e.g. “Name”) • help – help link (e.g. “return help();”) • divClass – override the default class • id – the element ID of the field • fieldSpecial – special options for the input field • required – if True, this is a required field • widget – the widget this applies to (use to pull values not provided) DASH page 20 WidgetPy - pwLink pwLink – display an HTML link Arguments: • text – the text of the link • link – the link target Keywords: • divClass – override the default class of “pwLink” • id – the element ID of the field • fieldSpecial – special options for the input field DASH page 21 WidgetPy - pwList pwList – display an list of widgets Arguments: • listItems – the list of widgets in this list Keywords: • divClass – override the default class of “pwList” • id – the element ID of the field • type – the type of list (bullet, number) • header – the list header DASH page 22 WidgetPy - pwListItem pwListItem – collect items for a pwList Arguments: • items – the list of widgets to include for this item Keywords: • divClass – override the default class of “pwListItem” • id – the element ID of the field • subitems – the subitems to include for this item DASH page 23 WidgetPy - pwError pwError – display an error with possible dialog processing Arguments: • text – the text of the error message Keywords: • caption – the field caption text (e.g. “Name”) • help – help link (e.g. “return help();”) • divClass – override the default class of “pwError” • id – the element ID of the field • dialogType – the dialog type (alert, confirm) • dialogText – the text for the dialog • okAction – what to do on an “OK” response • okArgs – arguments to pass to the “OK” action processor • cancelAction – what to do on a “Cancel” response • cancelArgs – arguments to pass to the “Cancel” action processor DASH page 24 Form Widgets pwText pwTextArea pwNumber pwFloat pwPhone pwDate pwSelector pwMultipleSelector pwStateSelector pwCountrySelector pwDoubleSelector pwCheckbox pwRadio pwButton pwFormValue DASH page 25 WidgetPy – pwText pwText – single line string input Arguments: • object – the object that contains the data • attribute – the name of the attribute with the text Keywords: • caption – the field caption text (e.g. “Name”) • help – help link (e.g. “return help();”) • example – the example text (e.g. “Thomas E. Ferrin”) • required – is this a required field or not • divClass – override the default class of “pwText” • id – the element ID of the field • fieldSpecial – special options for the input field • length – the length of the field (e.g. 32) • maxlength – the maximum number of characters (e.g. 32) • selectable – is this value selectable DASH page 26 WidgetPy – pwTextArea pwTextArea – multi-line string input Arguments: • object – the object that contains the data • attribute – the name of the attribute with the text Keywords: • caption – the field caption text (e.g. “Name”) • help – help link (e.g. “return help();”) • example – the example text (e.g. “Thomas E. Ferrin”) • required – is this a required field or not • divClass – override the default class of “pwTextArea” • id – the element ID of the field • fieldSpecial – special options for the input field • rows – the number of lines of text • columns – the maximum number of characters (e.g. 32) • selectable – is this value selectable DASH page 27 WidgetPy – pwNumber pwNumber – input an integer value Arguments: • object – the object that contains the data • attribute – the name of the attribute with the text Keywords: • caption – the field caption text (e.g. “Name”) • help – help link (e.g. “return help();”) • example – the example text (e.g. “100”) • required – is this a required field or not • divClass – override the default class of “pwNumber” • id – the element ID of the field • fieldSpecial – special options for the input field • length – the length of the field (e.g. 32) • selectable – is this value selectable • max – the maximum integer value allowed • min – the minimum integer value allowed DASH page 28 WidgetPy – pwFloat pwFloat – input a floating point value Arguments: • object – the object that contains the data • attribute – the name of the attribute with the text Keywords: • caption – the field caption text (e.g. “Name”) • help – help link (e.g. “return help();”) • example – the example text (e.g. “100”) • required – is this a required field or not • divClass – override the default class of “pwFloat” • id – the element ID of the field • fieldSpecial – special options for the input field • length – the length of the field (e.g. 32) • selectable – is this value selectable • max – the maximum floating point value allowed • min – the minimum floating point value allowed DASH page 29 WidgetPy – pwPhone pwPhone – input a phone number Arguments: • object – the object that contains the data • attribute – the name of the attribute with the text Keywords: • caption – the field caption text (e.g. “Name”) • help – help link (e.g. “return help();”) • example – the example text (e.g. “100”) • required – is this a required field or not • divClass – override the default class of “pwPhone” • id – the element ID of the field • fieldSpecial – special options for the input field • length – the length of the field (e.g. 32) • selectable – is this value selectable DASH page 30 WidgetPy – pwDate pwDate – input a date Arguments: • object – the object that contains the data • attribute – the name of the attribute with the text Keywords: • caption – the field caption text (e.g. “Name”) • help – help link (e.g. “return help();”) • example – the example text (e.g. “100”) • required – is this a required field or not • divClass – override the default class of “pwDate” • id – the element ID of the field • fieldSpecial – special options for the input field • length – the length of the field (e.g. 32) • selectable – is this value selectable • dateformat – a string representing the desired date format DASH page 31 WidgetPy – pwSelector pwSelector – select from a list of items Arguments: • object – the object that contains the data • attribute – the name of the attribute with the text • selectionList – list of (value,text) tuples where text will be displayed and value will be returned on selection Keywords: • caption – the field caption text (e.g. “Name”) • help – help link (e.g. “return help();”) • example – the example text (e.g. “Thomas E. Ferrin”) • required – is this a required field or not • divClass – override the default class of “pwSelector” • id – the element ID of the field • fieldSpecial – special options for the input field • multiple – if True, this is a multiple selector • header – if a multiple selector, this is the header • size – the number of rows to display • hiddenFields – if False, don’t output hidden fields for this widget • onclick – special “onclick” processing DASH page 32 WidgetPy – pwMultipleSelector pwMultipleSelector – select from a list of items Arguments: • object – the object that contains the data • attribute – the name of the attribute with the text • selectionList – list of (value,text) tuples where text will be displayed and value will be returned on selection Keywords: • caption – the field caption text (e.g. “Name”) • help – help link (e.g. “return help();”) • example – the example text (e.g. “Thomas E. Ferrin”) • required – is this a required field or not • divClass – override the default class of “pwSelector” • id – the element ID of the field • fieldSpecial – special options for the input field • header – the header for the multiple selector • size – the number of rows to display • hiddenFields – if False, don’t output hidden fields for this widget • onclick – special “onclick” processing DASH page 33 WidgetPy – pwStateSelector pwStateSelector – select from a list of states Arguments: • object – the object that contains the data • attribute – the name of the attribute with the text Keywords: • caption – the field caption text (e.g. “Name”) • help – help link (e.g. “return help();”) • example – the example text (e.g. “Thomas E. Ferrin”) • required – is this a required field or not • divClass – override the default class of “pwStateSelector” • id – the element ID of the field • fieldSpecial – special options for the input field • multiple – if True, this is a multiple selector • header – if a multiple selector, this is the header • size – the number of rows to display • hiddenFields – if False, don’t output hidden fields for this widget • onclick – special “onclick” processing DASH page 34 WidgetPy - pwCountrySelector pwCountrySelector – select from a list of countries Arguments: • object – the object that contains the data • attribute – the name of the attribute with the text Keywords: • caption – the field caption text (e.g. “Name”) • help – help link (e.g. “return help();”) • example – the example text (e.g. “Thomas E. Ferrin”) • required – is this a required field or not • divClass – override the default class of “pwCountrySelector” • id – the element ID of the field • fieldSpecial – special options for the input field • multiple – if True, this is a multiple selector • header – if a multiple selector, this is the header • size – the number of rows to display • hiddenFields – if False, don’t output hidden fields for this widget • onclick – special “onclick” processing DASH page 35 WidgetPy – pwDoubleSelector pwDoubleSelector – select from a list of items into a list Arguments: • object – the object that contains the data • attribute – the name of the attribute with the text • availList – list of (value,text) tuples (as in pwMultipleSelector) for the list of available items Keywords: • caption – the field caption text (e.g. “Name”) • help – help link (e.g. “return help();”) • example – the example text (e.g. “Thomas E. Ferrin”) • required – is this a required field or not • divClass – override the default class of “pwSelector” • id – the element ID of the field • fieldSpecial – special options for the input field • customEdit1 – custom “Edit” button. • customDelete1 – custom “Delete” button • customNew1 – custom “New” button 1Custom buttons allow for the modification of the default button actions. The value to a custom button is a dictionary of the form: {“label”:button label,”onclick”:New onclick action, “value”:value to return on select,”name”:button name} DASH page 36 WidgetPy – pwCheckbox pwCheckbox – a binary checkbox widget Arguments: • object – the object that contains the data • attribute – the name of the attribute with the text Keywords: • divClass – override the default class of “pwCheckbox” • id – the element ID of the field • fieldSpecial – special options for the input field • selectable – is this value selectable • type – type of checkbox (checkbox, radio) • label – label text • onclick – special “onclick” processing • name – name of the field for special form processing • checked – whether or not the checkbox is currently checked • labelPosition – the position of the label (top, bottom, left, right) • value – the value to return if the box is checked DASH page 37 WidgetPy – pwRadio pwRadio – a radio button widget Arguments: • object – the object that contains the data • attribute – the name of the attribute with the text • valueList – list of (value,text) tuples Keywords: • caption – the field caption text (e.g. “Name”) • help – help link (e.g. “return help();”) • example – the example text (e.g. “Thomas E. Ferrin”) • divClass – override the default class of “pwRadio” • required – is this a required field or not • id – the element ID of the field • fieldSpecial – special options for the input field • selectable – is this value selectable • labelPosition – the position of the label (top, bottom, left, right) DASH page 38 WidgetPy – pwButton pwButton – button widget Arguments: • None Keywords: • divClass – override the default class of “pwButton” • id – the element ID of the field • fieldSpecial – special options for the input field • selectable – is this value selectable • label – button label • image – link to the image for an image button • alt – alt text if an image button • onclick – special onclick processing • type – button type (submit) • value – value to return if this button is clicked • name – name of the button for special form processing DASH page 39 WidgetPy – pwFormValue pwFormValue – store values in a form using hidden fields Arguments: • object – the object that contains the data • attribute – the name of the attribute with the text Keywords: • divClass – override the default class of “pwFormValue” • id – the element ID of the field • fieldSpecial – special options for the input field • attrValue – value for this attribute to be restored by formProc DASH page 40 Grouping Widgets pwForm pwGroup pwTab pwHeader pwList pwListItem pwTable DASH page 41 WidgetPy – pwForm pwForm – collect a group of widgets into a form Arguments: • widgetList – a list of widgets Keywords: • divClass – override the default class of “pwForm” • id – the element ID of the field • fieldSpecial – special options for the input field • method – submission method (POST, GET) • action – the submission url • onsubmit – any special submission processing DASH page 42 WidgetPy – pwGroup pwGroup – collect a group of widgets into a group Arguments: • widgetList – a list of widgets Keywords: • caption – the field caption text (e.g. “Name”) • help – help link (e.g. “return help();”) • divClass – override the default class of “pwGroup” • id – the element ID of the field • fieldSpecial – special options for the input field • collapse – if True this is a collapsing Group • collapsed – if True this collapsing group is collapsed • obj – object to track collapse state • attr – attribute in “obj” to track collapse state DASH page 43 WidgetPy – pwTab pwTab – collect a group of widgets into a tab set Arguments: • tabList – list of tuples of the form (tab-label, widgetList) Keywords: • help – help link (e.g. “return help();”) • divClass – override the default class of “pwForm” • id – the element ID of the field • fieldSpecial – special options for the input field • selected – the label of the tab currently selected • disabled – list of the labels of the currently disabled tabs DASH page 44 WidgetPy -- pwHeader pwHeader – display the default HTML header Arguments: • widgetList – a list of widgets Keywords: • title – Title for the HTML <title> tag • logo – link to possible logo • logoLink – where to go if the logo gets “clicked” • extraStyle – additional stylesheets passed as a list of tuples of the form (stylesheet, label). Stylesheets without labels are assumed to be extra stylesheets rather than alternate • extraJS – links to additional javascript processing files • banner – HTML banner DASH page 45 WidgetPy - pwList pwList – display an list of widgets Arguments: • listItems – the list of widgets in this list Keywords: • divClass – override the default class of “pwList” • id – the element ID of the field • type – the type of list (bullet, number) • header – the list header DASH page 46 WidgetPy - pwListItem pwListItem – collect items for a pwList Arguments: • items – the list of widgets to include for this item Keywords: • divClass – override the default class of “pwListItem” • id – the element ID of the field • subitems – the subitems to include for this item DASH page 47 WidgetPy – pwTable pwTable – collect a group of widgets or values into a table Arguments: • data – a list of lists of values. Value can be either text values or widgets Keywords: • divClass – override the default class of “pwTable” • id – the element ID of the field • fieldSpecial – special options for the input field • rowlabels – list of lists of row labels • colheaders – list of lists of column headers • caption – table caption • rowalign – list of lists of vertical alignment values • colalign – list of lists of horizontal alignment values DASH page 48 Other Methods DASH page 49 WidgetPy – formProc formProc – process a form widget Arguments: • formData – form data element from Python cgi handler • ctorDict – dictionary of object constructors of the form {“name”:constructor} Returns: • Dictionary of resulting objects of the form: {“name<id#>”:object} DASH page 50 Additional Examples Tabs & Stylesheet Switching DASH page 51 Additional Examples Tabular Input DASH page 52 DASH DAta SHaring Infrastructure DASH page 53 Outline Problem Statement Event Model DASH Description DASH page 54 Problem Statement • Biological Science is increasingly a collaborative science • Much of this collaboration involves the dissemination and sharing of data between disparate groups with disparate disciplines - Often small to medium sized labs - Often very different computing environments and levels of IT expertise • Traditionally, this data sharing has been labor-intensive and error-prone - Data conversion issues - Need for automation DASH page 55 Problem Statement • Collaboration is critical, but so are the issues of data ownership and integrity - Desire to protect data until its “ready” • Data processing is often necessarily ad-hoc - New results might require different processing approaches - Different researchers might process results differently for different purposes • Existing approaches to this problem not sufficient - Filesharing approaches require compatible computing environments - Workflow approaches require significant investment in process design - Grid approaches require compatible computing environments and a level of shared security infrastructure DASH page 56 Trim sequence data socr:> ls Lskdjflskjfdlkfj lsdkjfdlj Lskdjflsk jlksdjfls kdj]l Sldkjsldj Lsjkfldj Lsjflskjdfkjdjdklj Lskjfldfj Lksdjfldkjf Putative gene id for trap Blast against genomic data Create .png image socr:> ls Lskdjflskjfdlkfj lsdkjfdlj Lskdjflsk jlksdjfls kdj]l Sldkjsldj Lsjkfldj Lsjflskjdfkjdjdklj Lskjfldfj Lksdjfldkjf Blast against GenBank NR Populate database socr:> ls Lskdjflskjfdlkfj lsdkjfdlj Lskdjflsk jlksdjfls kdj]l Sldkjsldj Lsjkfldj Lsjflskjdfkjdjdklj Lskjfldfj Lksdjfldkjf Genomics Core Facility Gene trapping experiments Sequencing socr:> ls Lskdjflskjfdlkfj lsdkjfdlj Lskdjflsk jlksdjfls kdj]l Sldkjsldj Lsjkfldj Lsjflskjdfkjdjdklj Lskjfldfj Lksdjfldkjf socr:> ls Lskdjflskjfdlkfj lsdkjfdlj Lskdjflsk jlksdjfls kdj]l Sldkjsldj Lsjkfldj Lsjflskjdfkjdjdklj Lskjfldfj Lksdjfldkjf Reject bad sequences socr:> ls Lskdjflskjfdlkfj lsdkjfdlj Lskdjflsk jlksdjfls kdj]l Sldkjsldj Lsjkfldj Lsjflskjdfkjdjdklj Lskjfldfj Lksdjfldkjf Confirmation of id socr:> ls Lskdjflskjfdlkfj lsdkjfdlj Lskdjflsk jlksdjfls kdj]l Sldkjsldj Lsjkfldj Lsjflskjdfkjdjdklj Lskjfldfj Lksdjfldkjf Update web page Refined sequence data PNG socr:> ls Lskdjflskjfdlkfj lsdkjfdlj Lskdjflsk jlksdjfls kdj]l Sldkjsldj Lsjkfldj Lsjflskjdfkjdjdklj Lskjfldfj Lksdjfldkjf Create .html files HTML HTML Sequence data 40 – 500 bp Example: Baygenomics DASH page 57 Solution: The Event Model Event model is based on responses to events • Events could include file updates, database record updates, e-mail delivery, web service messages, web posts, time The response to an event depends on an event handler • Event handlers are programs designed to deal with a specific event • Can have multiple event handlers for a single event Events can be combined logically • Boolean operations of AND, OR, and NOT, can be used to create “virtual events” to trigger specific event handlers DASH page 58 DASH’s Event Model DASH utilizes the event model Currently supported event types: • File update - E-Mail is handled through file update • Database record update Currently supported Boolean combinations: • OR, AND, NOT Event Handlers: • Similar to UNIX filters • Take an input file and transform it to an output file • Output file can be used to trigger additional event handlers - Can be explicit or implicit • Security is handled utilizing native security (file system or database) Event registration • DASH events and event handlers are registered in a MySQL Database DASH page 59 Why Use an Event Model ? Advantages • Multiple events can be combined sequentially to create a data flow • Events are fine-grained, so lend themselves to ad-hoc definition and re-definition • Events don’t depend on underlying infrastructure - Can utilize data sharing technologies, database technologies, Web Services, E-Mail or FTP to trigger an event - Provides for a multitude of underlying mechanisms • Event approach does not assume any specific shared security realm - Triggering an event does not imply any global security - Security for objects (files, records, etc.) can be fine-grained Disadvantages • Does not explicitly deal with data delivery • Does not provide for process-centric validation and control - Not appropriate for business DASH page 60 DASH Conceptual Architecture File System Event Monitor Protocol Initiation DASH Event Registry Event Registration Database DASH File System Poller Database Poller socr:> ls Lskdjflskjfdlkfj lsdkjfdlj Lskdjflsk jlksdjfls kdj]l Sldkjsldj Lsjkfldj Lsjflskjdfkjdjdklj Lskjfldfj Lksdjfldkjf Processes Time Process Status Time Events File Update Record Update Process Failure Process Completion Clock socr:> ls Lskdjflskjfdlkfj lsdkjfdlj Lskdjflsk jlksdjfls kdj]l Sldkjsldj Lsjkfldj Lsjflskjdfkjdjdklj Lskjfldfj Lksdjfldkjf Processes Events Configuration information Web Services SOAP Events SOAP Messag e Web Services DASH page 61 DASH Event Model Event Dispatcher Event Handler Invoke Supplied by DASH Normally supplied by user File System Database … ? ? ? Event DASH Building Block File System Database socr:> ls Lskdjflskjfdlkfj lsdkjfdlj Lskdjflsk jlksdjfls kdj]l Sldkjsldj Lsjkfldj Lsjflskjdfkjdjdklj Lskjfldfj Lksdjfldkjf Processes Time Web Services Event Action Output DASH page 62 DASH Event Model – Pipelines Event Event Dispatcher Event Handler Invoke Output Protocol 1 Event Dispatcher Event Event Handler Invoke Output Protocol 2 Event Handler Event Dispatcher Event Handler Event Dispatcher Event Handler Event Dispatcher Event Handler Event Dispatcher Event Handler Event Dispatcher DASH page 63 DASH Event Model – Virtual Events Event Handler Event Dispatcher Event Handler Event Dispatcher Event Handler Event Dispatcher Event Handler Event Dispatcher Event Handler Event Dispatcher Events can be combined using Boolean operations: • AND • OR • NOT Event Handler Event Dispatcher AND Event Handler Event Dispatcher Event Handler Event Dispatcher Event Handler Event Dispatcher Event Handler Event Dispatcher Event Handler Event Dispatcher Event Handler Event Dispatcher OR DASH page 64 DASH Event Model – Patterns One standard event pattern: • IF a file has been updated, • AND a certain process has completed • THEN start the next process Similar to steps in a Data Flow Diagram File 1 File 1 socr:> ls Lskdjflskjfdlkfj lsdkjfdlj Lskdjflsk jlksdjfls kdj]l Sldkjsldj Lsjkfldj Lsjflskjdfkjdjdklj Lskjfldfj Lksdjfldkjf Protocol 1 complete File 2 File 2 Event Handler Event Dispatcher Protocol 1 Event Handler Event Dispatcher AND socr:> ls Lskdjflskjfdlkfj lsdkjfdlj Lskdjflsk jlksdjfls kdj]l Sldkjsldj Lsjkfldj Lsjflskjdfkjdjdklj Lskjfldfj Lksdjfldkjf Protocol 2 complete File 3 File 3 Protocol 2 DASH page 65 Genomics Core Facility Gene trapping experiments Sequencing Trim sequence data Blast against GenBank NR Populate database Putative gene id for trap Sequence data 40 – 500 bp Blast against genomic data Create .png image Reject bad sequences socr:> ls Lskdjflskjfdlkfj lsdkjfdlj Lskdjflsk jlksdjfls kdj]l Sldkjsldj Lsjkfldj Lsjflskjdfkjdjdklj Lskjfldfj Lksdjfldkjf socr:> ls Lskdjflskjfdlkfj lsdkjfdlj Lskdjflsk jlksdjfls kdj]l Sldkjsldj Lsjkfldj Lsjflskjdfkjdjdklj Lskjfldfj Lksdjfldkjf socr:> ls Lskdjflskjfdlkfj lsdkjfdlj Lskdjflsk jlksdjfls kdj]l Sldkjsldj Lsjkfldj Lsjflskjdfkjdjdklj Lskjfldfj Lksdjfldkjf socr:> ls Lskdjflskjfdlkfj lsdkjfdlj Lskdjflsk jlksdjfls kdj]l Sldkjsldj Lsjkfldj Lsjflskjdfkjdjdklj Lskjfldfj Lksdjfldkjf socr:> ls Lskdjflskjfdlkfj lsdkjfdlj Lskdjflsk jlksdjfls kdj]l Sldkjsldj Lsjkfldj Lsjflskjdfkjdjdklj Lskjfldfj Lksdjfldkjf socr:> ls Lskdjflskjfdlkfj lsdkjfdlj Lskdjflsk jlksdjfls kdj]l Sldkjsldj Lsjkfldj Lsjflskjdfkjdjdklj Lskjfldfj Lksdjfldkjf socr:> ls Lskdjflskjfdlkfj lsdkjfdlj Lskdjflsk jlksdjfls kdj]l Sldkjsldj Lsjkfldj Lsjflskjdfkjdjdklj Lskjfldfj Lksdjfldkjf socr:> ls Lskdjflskjfdlkfj lsdkjfdlj Lskdjflsk jlksdjfls kdj]l Sldkjsldj Lsjkfldj Lsjflskjdfkjdjdklj Lskjfldfj Lksdjfldkjf Confirmation of id Update web page Create .html files Refined sequence data PNG HTML HTML A Closer Look DASH page 66 socr:> ls Lskdjflskjfdlkfj lsdkjfdlj Lskdjflsk jlksdjfls kdj]l Sldkjsldj Lsjkfldj Lsjflskjdfkjdjdklj Lskjfldfj Lksdjfldkjf DASH Event Dispatcher match against event registry….. DASH file generator socr:> ls Lskdjflskjfdlkfj lsdkjfdlj Lskdjflsk jlksdjfls kdj]l Sldkjsldj Lsjkfldj Lsjflskjdfkjdjdklj Lskjfldfj Lksdjfldkjf file modification event Refined sequence data Blast against GenBank NR Create .html files Putative gene id event DASH page 67 socr:> ls Lskdjflskjfdlkfj lsdkjfdlj Lskdjflsk jlksdjfls kdj]l Sldkjsldj Lsjkfldj Lsjflskjdfkjdjdklj Lskjfldfj Lksdjfldkjf DASH Event Dispatcher invoke registered handlers socr:> ls Lskdjflskjfdlkfj lsdkjfdlj Lskdjflsk jlksdjfls kdj]l Sldkjsldj Lsjkfldj Lsjflskjdfkjdjdklj Lskjfldfj Lksdjfldkjf DASH process handler start process Refined sequence data Blast against GenBank NR Create .html files Putative gene id event DASH page 68 socr:> ls Lskdjflskjfdlkfj lsdkjfdlj Lskdjflsk jlksdjfls kdj]l Sldkjsldj Lsjkfldj Lsjflskjdfkjdjdklj Lskjfldfj Lksdjfldkjf DASH Event Dispatcher match against event registry… socr:> ls Lskdjflskjfdlkfj lsdkjfdlj Lskdjflsk jlksdjfls kdj]l Sldkjsldj Lsjkfldj Lsjflskjdfkjdjdklj Lskjfldfj Lksdjfldkjf DASH process handler DASH database generator AND DASH sync. generator database modified event process complete event Refined sequence data Blast against GenBank NR Create .html files Putative gene id event sync event DASH page 69 Refined sequence data Blast against GenBank NR socr:> ls Lskdjflskjfdlkfj lsdkjfdlj Lskdjflsk jlksdjfls kdj]l Sldkjsldj Lsjkfldj Lsjflskjdfkjdjdklj Lskjfldfj Lksdjfldkjf DASH Event Dispatcher invoke registered handlers socr:> ls Lskdjflskjfdlkfj lsdkjfdlj Lskdjflsk jlksdjfls kdj]l Sldkjsldj Lsjkfldj Lsjflskjdfkjdjdklj Lskjfldfj Lksdjfldkjf DASH process handler Create .html files Putative gene id start process event DASH page 70 sync event socr:> ls Lskdjflskjfdlkfj lsdkjfdlj Lskdjflsk jlksdjfls kdj]l Sldkjsldj Lsjkfldj Lsjflskjdfkjdjdklj Lskjfldfj Lksdjfldkjf DASH Event Dispatcher match against event registry… socr:> ls Lskdjflskjfdlkfj lsdkjfdlj Lskdjflsk jlksdjfls kdj]l Sldkjsldj Lsjkfldj Lsjflskjdfkjdjdklj Lskjfldfj Lksdjfldkjf DASH process handler DASH file handler AND DASH sync. generator process complete event DASH sync. generator OR file modified event Refined sequence data Blast against GenBank NR Putative gene id Create .html files event DASH page 71 Refined sequence data Blast against GenBank NR Create .html files Putative gene id socr:> ls Lskdjflskjfdlkfj lsdkjfdlj Lskdjflsk jlksdjfls kdj]l Sldkjsldj Lsjkfldj Lsjflskjdfkjdjdklj Lskjfldfj Lksdjfldkjf DASH Event Dispatcher invoke registered handlers socr:> ls Lskdjflskjfdlkfj lsdkjfdlj Lskdjflsk jlksdjfls kdj]l Sldkjsldj Lsjkfldj Lsjflskjdfkjdjdklj Lskjfldfj Lksdjfldkjf ??? event DASH page 72 DASH Futures New Event Types: • Web Service (SOAP) messages • Web CGI posts DASH protocol documentation • Export of DASH protocols using Web Services Choreography Description Language • Discovery of implicit protocols using Web Services Ease-of-use improvements • Web-based registration of events and handlers • Web-based monitoring of events and handler progress • Provided wrappers for common applications • “Generic” wrapper to quickly wrap commercial applications - Requires a command-line interface Extensibility • Use of WSDL and SOAP to extend DASH events across a network DASH page 73 Questions ? DASH page 74 Workflow Model Workflow systems utilize a process-centric model • Significant investment in process design • Processes explicitly involve external steps (e.g. approval) • Most implementations are very business-centric Workflow systems are very appropriate for business • Processes are well-defined and stable • Need for documentation and agreement of process implementations • Up-front investment in process design has significant payback Example systems: • BEA, Taverna, TIBCO DASH page 75 Workflow Model Advantages • Rigorous process-oriented definition • Substantial tool support for process definition • Well-adapted to business where processes need to be reviewed Disadvantages • Definition process generally requires dedicated resources - Process analysts • Inhibits ad-hoc definitions • Integration is often very expensive • Generally centralized DASH page 76 Data Sharing Model Data sharing approaches provide for data availability • All data available, all the time • Requires substantial agreements about formats and processing rules • Assumes some level of shared security realm • Requires a level of system compatibility • Potential for significant management expenses Example systems: • Global File System (GFS), NFS, Avaki, Lustre, Centralized Databases DASH page 77 Data Sharing Model Advantages • Provides for globally shared data • Data is available “instantaneously” to all participating users • Some systems provide for built-in data conversion (e.g. Avaki) Disadvantages • Must have standardized infrastructure - Packages may not work on all systems • Significant implementation overhead • Implies shared (or at least agreed-to) security realms • No provision for workflow DASH page 78 Batch Management Systems Batch management systems support the queuing and management of processes • Often across multiple systems (distributed) • Sometimes utilizing underlying distributed libraries (computational clusters, Grid) Role of a batch management system is to efficiently manage computational resources • Explicit goal is to increase throughput Batch management systems often include some facility for handling input and output files Example systems: • Platform (LSF, Symphony), NQS, PBS, Globus, SunGrid DASH page 79 Batch Management Systems Advantages • Efficiently manages processes across distributed computers Disadvantages • No facility to trigger off of data-driven events • Generally requires broad integration and cooperation amongst all participants
