OGF PGI - Requirements for Job Description
Morris, Johannes and all, Concerning the OGF PGI requirements for Job Description : From the EDGI use case 'Marshal Activities from Service Grid to Desktop Grid', the most important requirements, sorted by decreasing priority, are : JD3 (100) The Job Description document MUST reference all grid entities in conformance to the GLUE specification JD11 (107) Job Description should be only used for Job Description not for any kind of feedback. Job status information should not be communicated with a 'changed JSDL' JD4 (101) The Job Description document specification MUST permit the Client to request automatic data stage-in and stage-out JD22 (116) Clearly separate file and directory in the data-stagings. e.g. within the job description the element should be clearly labeled as file or directory, e.g. different aspect of handling directory or a file The other requirements labeled 'yes' are welcome, EXCEPT the 2 following ones, which generate too much COMPLEXITY in the Execution Service : JD6 (103) The Job Description document specification MUST permit the Client to request that at specified Activity states, the Execution service sets the Activity on 'Hold' [NO] JD8 (105) Job Description supports, 1) service-directed data-staging (i.e. specify data-stagings),2) client-initiated data-staging (i.e. specifiy data-staging but service will do nothing only control whether files existing), 3) manual data-staging (i.e. no specification of data-staging elements) [NO] Best regards. ----------------------------------------------------- Etienne URBAH LAL, Univ Paris-Sud, IN2P3/CNRS Bat 200 91898 ORSAY France Tel: +33 1 64 46 84 87 Skype: etienne.urbah Mob: +33 6 22 30 53 27 mailto:urbah@lal.in2p3.fr -----------------------------------------------------
Morris, Johannes and all, Concerning the OGF PGI requirements for Job Description : I have updated the Wiki page at http://forge.gridforum.org/sf/wiki/do/viewPage/projects.pgi-wg/wiki/Requirem... and the detailed pages for Job Description according to Amsterdam meeting 28-29 April 2010. So you can add you own comments and suggestions in the detail pages. If you find any inaccuracy, please correct it directly or add a comment in the appropriate detail page. Best regards. ----------------------------------------------------- Etienne URBAH LAL, Univ Paris-Sud, IN2P3/CNRS Bat 200 91898 ORSAY France Tel: +33 1 64 46 84 87 Skype: etienne.urbah Mob: +33 6 22 30 53 27 mailto:urbah@lal.in2p3.fr ----------------------------------------------------- On Thu, 30/09/2010 20:14, Etienne URBAH wrote:
Morris, Johannes and all,
Concerning the OGF PGI requirements for Job Description :
From the EDGI use case 'Marshal Activities from Service Grid to Desktop Grid', the most important requirements, sorted by decreasing priority, are :
JD3 (100) The Job Description document MUST reference all grid entities in conformance to the GLUE specification
JD11 (107) Job Description should be only used for Job Description not for any kind of feedback. Job status information should not be communicated with a 'changed JSDL'
JD4 (101) The Job Description document specification MUST permit the Client to request automatic data stage-in and stage-out
JD22 (116) Clearly separate file and directory in the data-stagings. e.g. within the job description the element should be clearly labeled as file or directory, e.g. different aspect of handling directory or a file
The other requirements labeled 'yes' are welcome, EXCEPT the 2 following ones, which generate too much COMPLEXITY in the Execution Service :
JD6 (103) The Job Description document specification MUST permit the Client to request that at specified Activity states, the Execution service sets the Activity on 'Hold' [NO]
JD8 (105) Job Description supports, 1) service-directed data-staging (i.e. specify data-stagings),2) client-initiated data-staging (i.e. specifiy data-staging but service will do nothing only control whether files existing), 3) manual data-staging (i.e. no specification of data-staging elements) [NO]
Best regards.
----------------------------------------------------- Etienne URBAH LAL, Univ Paris-Sud, IN2P3/CNRS Bat 200 91898 ORSAY France Tel: +33 1 64 46 84 87 Skype: etienne.urbah Mob: +33 6 22 30 53 27 mailto:urbah@lal.in2p3.fr -----------------------------------------------------
Morris, Johannes and all, Concerning the OGF PGI requirements for Job Description : Each requirement for Job Description implicitly contains one or several underlying requirements for the Execution Service. In order to ease mutual understanding, I suggest, for each requirement for Job Description, to answer to following questions in a precise order : A) What is the precise underlying requirement for the Execution Service ? Focusing on this underlying requirement for the Execution Service : B) Is it really needed ? C) How much does the current JSDL recommendation permit to express it ? What is the gap ? D) Is it possible to conceptually define an extension of the current JSDL recommendation in order to express it ? (NO XML at this step) E) If not, what is the conceptual proposal for a modification of the current JSDL recommendation in order to express it ? (NO XML at this step) F) Proposal of an example written in XML. For each requirement, we do NOT need to answer all questions immediately, but we need to answer them in this precise order, and in particular NOT to begin with an XML example (which is implementation detail). Best regards. ----------------------------------------------------- Etienne URBAH LAL, Univ Paris-Sud, IN2P3/CNRS Bat 200 91898 ORSAY France Tel: +33 1 64 46 84 87 Skype: etienne.urbah Mob: +33 6 22 30 53 27 mailto:urbah@lal.in2p3.fr ----------------------------------------------------- On Mon, 04/10/2010 18:41, Etienne URBAH wrote:
Morris, Johannes and all,
Concerning the OGF PGI requirements for Job Description :
I have updated the Wiki page at http://forge.gridforum.org/sf/wiki/do/viewPage/projects.pgi-wg/wiki/Requirem... and the detailed pages for Job Description according to Amsterdam meeting 28-29 April 2010.
So you can add you own comments and suggestions in the detail pages.
If you find any inaccuracy, please correct it directly or add a comment in the appropriate detail page.
Best regards.
----------------------------------------------------- Etienne URBAH LAL, Univ Paris-Sud, IN2P3/CNRS Bat 200 91898 ORSAY France Tel: +33 1 64 46 84 87 Skype: etienne.urbah Mob: +33 6 22 30 53 27 mailto:urbah@lal.in2p3.fr -----------------------------------------------------
On Thu, 30/09/2010 20:14, Etienne URBAH wrote:
Morris, Johannes and all,
Concerning the OGF PGI requirements for Job Description :
From the EDGI use case 'Marshal Activities from Service Grid to Desktop Grid', the most important requirements, sorted by decreasing priority, are :
JD3 (100) The Job Description document MUST reference all grid entities in conformance to the GLUE specification
JD11 (107) Job Description should be only used for Job Description not for any kind of feedback. Job status information should not be communicated with a 'changed JSDL'
JD4 (101) The Job Description document specification MUST permit the Client to request automatic data stage-in and stage-out
JD22 (116) Clearly separate file and directory in the data-stagings. e.g. within the job description the element should be clearly labeled as file or directory, e.g. different aspect of handling directory or a file
The other requirements labeled 'yes' are welcome, EXCEPT the 2 following ones, which generate too much COMPLEXITY in the Execution Service :
JD6 (103) The Job Description document specification MUST permit the Client to request that at specified Activity states, the Execution service sets the Activity on 'Hold' [NO]
JD8 (105) Job Description supports, 1) service-directed data-staging (i.e. specify data-stagings),2) client-initiated data-staging (i.e. specifiy data-staging but service will do nothing only control whether files existing), 3) manual data-staging (i.e. no specification of data-staging elements) [NO]
Best regards.
----------------------------------------------------- Etienne URBAH LAL, Univ Paris-Sud, IN2P3/CNRS Bat 200 91898 ORSAY France Tel: +33 1 64 46 84 87 Skype: etienne.urbah Mob: +33 6 22 30 53 27 mailto:urbah@lal.in2p3.fr -----------------------------------------------------
Morris, Johannes and all, Please find below the OGF PGI requirements triggered by the EDGI Use Cases : +-----------------------------+ | A) Mandatory requirements | +-----------------------------+ Following is the list of requirements with 'yes' status which are seen as mandatory for EDGI. Interoperation would be very difficult without them. Information functionality ------------------------- 1 : All grid entities (if possible) MUST be described using the GLUE model. If not possible, extensions for the GLUE model are necessary. http://forge.gridforum.org/sf/wiki/do/viewPage/projects.pgi-wg/wiki/ReqIS1 5 : The Execution Service MUST NOT expose detailed information about the GLUE entities which the Execution Service does not manage (all that are not expressed by the computing part of GLUE). For example, Storage Element GLUE entity NOT exposed by Execution Service, NO details about Storage entity. http://forge.gridforum.org/sf/wiki/do/viewPage/projects.pgi-wg/wiki/ReqIS4 165 : For already finished Activities, the Information functionality SHOULD accept requests querying the Activity Status, but MAY return an error http://forge.gridforum.org/sf/wiki/do/viewPage/projects.pgi-wg/wiki/ReqIS9 172 : In order to prevent overloading the Execution Service (which has performance requirements for Activity Management), the Information functionality SHOULD be separated from the Execution Service http://forge.gridforum.org/sf/wiki/do/viewPage/projects.pgi-wg/wiki/ReqIS14 Security (Authentication, Authorization, Delegation) ---------------------------------------------------- 9 : If server authentication to clients, then it must be done with X.509 certificates on TLS (as mandatory option, allowing also other mechanisms additionally) http://forge.gridforum.org/sf/wiki/do/viewPage/projects.pgi-wg/wiki/ReqAA1 11 : Each Service MUST publish the Authentication and Authorization methods accepted by its Endpoints in conformance with GLUE recommendations http://forge.gridforum.org/sf/wiki/do/viewPage/projects.pgi-wg/wiki/ReqAA2 32 : There must be a mechanism which allows users to manage Activities submitted by other users (access control lists/methods/policies). In order to authorize (or not) an request on an Activity, each instance of the Execution Service MUST enforce a consistent authorization method. http://forge.gridforum.org/sf/wiki/do/viewPage/projects.pgi-wg/wiki/ReqAA9 Application Repository ---------------------- 34 : An easy way of launching applications or pre-configured /pre-installed software w/o specifying location details. Installed/pre-configured ones should be exposed as well as part of the resource description. http://forge.gridforum.org/sf/wiki/do/viewPage/projects.pgi-wg/wiki/ReqAR1 Accounting ---------- 36 : The Execution Service SHOULD provide Accounting records for each managed Activity. e.g. OGF Usage Records, for tracking resource usage. Most likely improving UR in terms of network and storage resource tracking and improvements of compute parts of multi-core-business. Grid-level attributes (for example: start-time on Grid vs. in batch-system). http://forge.gridforum.org/sf/wiki/do/viewPage/projects.pgi-wg/wiki/ReqAc2 173 : In order to prevent overloading the Execution Service (which has performance requirements for Activity Management), the Accounting functionality MUST be separated from the Execution Service http://forge.gridforum.org/sf/wiki/do/viewPage/projects.pgi-wg/wiki/ReqAc1 Logging and Bookkeeping ----------------------- 37 : WITH ITS ORIGINAL TITLE : In order to permit efficient Log analysis and Security audit, some Logging and Bookkeeping functionality MUST secure persistence at the grid level of Activity logs from various logging systems and MUST permit Client access to these Activity logs, even after Activities have finished. http://forge.gridforum.org/sf/wiki/do/viewPage/projects.pgi-wg/wiki/ReqLB1 Activity Management ------------------- 45 : On creation of an Activity, the Execution Service MUST return to the Client an Activity ID permitting the Client to perform subsequent actions (Query, Cancel, ...) on this precise Activity http://forge.gridforum.org/sf/wiki/do/viewPage/projects.pgi-wg/wiki/ReqJM4 62 : The Execution Service MUST log enough grid information inside logging systems (which MAY vary during the lifetime of the Activity), in order to permit efficient Log analysis and Security audit http://forge.gridforum.org/sf/wiki/do/viewPage/projects.pgi-wg/wiki/ReqJM6 67 : If a Client queries for an already finished Activity, the Execution Service MAY response with an error http://forge.gridforum.org/sf/wiki/do/viewPage/projects.pgi-wg/wiki/ReqJM6.8 74 : The execution service MUST NOT expose activity information when queried for resource information http://forge.gridforum.org/sf/wiki/do/viewPage/projects.pgi-wg/wiki/ReqJM6.1... 94 : Requirement to have a purge (maybe called wipe) operation. Removing all presence (except logs & usage records) of the activity (when it is not longer active or once finished). This functionality is only allowed on any final state according to the PGI state model. The functionality is not supposed to kill Activities, that is why its only allowed on final states. http://forge.gridforum.org/sf/wiki/do/viewPage/projects.pgi-wg/wiki/ReqJM20 Job Description --------------- 100 : The Job Description document MUST reference all grid entities in conformance to the GLUE specification http://forge.gridforum.org/sf/wiki/do/viewPage/projects.pgi-wg/wiki/ReqJD3 101 : The Job Description document specification MUST permit the Client to request automatic data stage-in and stage-out http://forge.gridforum.org/sf/wiki/do/viewPage/projects.pgi-wg/wiki/ReqJD4 107 : Job Description should be only used for Job Description not for any kind of feedback. Job status information should not be communicated with a 'changed JSDL' http://forge.gridforum.org/sf/wiki/do/viewPage/projects.pgi-wg/wiki/ReqJD11 State Model ----------- 119 : The Execution service MUST support a common state model http://forge.gridforum.org/sf/wiki/do/viewPage/projects.pgi-wg/wiki/ReqSM1 131 : The Execution service MUST perform the transition between Activity states requested by the Client ONLY if it makes sense. Otherwise, the Execution service MUST return an error to the Client. http://forge.gridforum.org/sf/wiki/do/viewPage/projects.pgi-wg/wiki/ReqSM10.... Data Management --------------- NO requirement with 'yes' status is essential. Non functional -------------- 146 : Software components (Services and Clients) MUST ease traceability of the original author of a request http://forge.gridforum.org/sf/wiki/do/viewPage/projects.pgi-wg/wiki/ReqNF3 147 : Software components (Services and Clients) MUST generate adequate logs http://forge.gridforum.org/sf/wiki/do/viewPage/projects.pgi-wg/wiki/ReqNF4 148 : Software components (Services and Clients) MUST generate and propagate meaningful error messages, including context description http://forge.gridforum.org/sf/wiki/do/viewPage/projects.pgi-wg/wiki/ReqNF5 149 : Specifications SHOULD prevent the occurrence of SPOFs and bottlenecks http://forge.gridforum.org/sf/wiki/do/viewPage/projects.pgi-wg/wiki/ReqNF7 154 : Basic SW Engineering basic principles - implementation encapsulation, separation of policies, construction by composition, don't re-invent new mechanisms when some existing http://forge.gridforum.org/sf/wiki/do/viewPage/projects.pgi-wg/wiki/ReqNF12 160 : Execution Service MUST NOT try to provide ALL grid functionalities, but MUST have a well defined scope, and MUST work together with other grid services : Security, Accounting, Data (Storage, Movement, Catalog, ...), perhaps Information, Logging and Bookkeeping http://forge.gridforum.org/sf/wiki/do/viewPage/projects.pgi-wg/wiki/ReqNF13 +------------------------------------+ | B) Complete list of requirements | +------------------------------------+ Following is the complete list of requirements with 'yes' status which are seen as useful for EDGI. Information functionality : 1, 3, 4, 5, 6, 8, 165, 166, 172 Security (Authentication, Authorization, Delegation) : 9, 11, 17, 19, 20, 21, 22, 32 Application Repository : 34 Accounting : 36, 173 Logging and Bookkeeping : 37 WITH ITS ORIGINAL TITLE Activity Management : 43, 44, 45, 46, 55, 58, 59, 62, 63, 64, 65, 66, 67, 68, 74, 81, 82, 86, 90, 94 Job Description : 100, 101, 104, 107, 112, 114, 116, 118 State Model : 119, 120, 121, 122, 123, 124, 125, 128, 131, 134 Data Management : 143 Non functional : 145, 146, 147, 148, 149, 150, 154, 156, 159, 160 Best regards. ----------------------------------------------------- Etienne URBAH LAL, Univ Paris-Sud, IN2P3/CNRS Bat 200 91898 ORSAY France Tel: +33 1 64 46 84 87 Skype: etienne.urbah Mob: +33 6 22 30 53 27 mailto:urbah@lal.in2p3.fr ----------------------------------------------------- On Thu, 30/09/2010 20:14, Etienne URBAH wrote:
Morris, Johannes and all,
Concerning the OGF PGI requirements for Job Description :
From the EDGI use case 'Marshal Activities from Service Grid to Desktop Grid', the most important requirements, sorted by decreasing priority, are :
JD3 (100) The Job Description document MUST reference all grid entities in conformance to the GLUE specification
JD11 (107) Job Description should be only used for Job Description not for any kind of feedback. Job status information should not be communicated with a 'changed JSDL'
JD4 (101) The Job Description document specification MUST permit the Client to request automatic data stage-in and stage-out
JD22 (116) Clearly separate file and directory in the data-stagings. e.g. within the job description the element should be clearly labeled as file or directory, e.g. different aspect of handling directory or a file
The other requirements labeled 'yes' are welcome, EXCEPT the 2 following ones, which generate too much COMPLEXITY in the Execution Service :
JD6 (103) The Job Description document specification MUST permit the Client to request that at specified Activity states, the Execution service sets the Activity on 'Hold' [NO]
JD8 (105) Job Description supports, 1) service-directed data-staging (i.e. specify data-stagings),2) client-initiated data-staging (i.e. specifiy data-staging but service will do nothing only control whether files existing), 3) manual data-staging (i.e. no specification of data-staging elements) [NO]
Best regards.
----------------------------------------------------- Etienne URBAH LAL, Univ Paris-Sud, IN2P3/CNRS Bat 200 91898 ORSAY France Tel: +33 1 64 46 84 87 Skype: etienne.urbah Mob: +33 6 22 30 53 27 mailto:urbah@lal.in2p3.fr -----------------------------------------------------
participants (1)
-
Etienne URBAH