July 2005 - saga-rg - lists.ogf.org

Current issue list 21/07/05
by Tom Goodale 21 Jul '05

21 Jul '05

--------------------------------------------------------------------- Peter Kunszt <Peter.Kunszt(a)cern.ch> Kalman Kovari <Kalman.Kovari(a)cern.ch> - The examples are sometimes more misleading than useful... -> why? -> TODO ANDRE: clearify, discuss on list - The error handling and error semantics are not defined well enough. This is essential in order to assure proper interoperability. -> compile list of error codes, check for consistency -> TODO TOM: create proposal for procedural and OO - There is a lack of structure in terms of interface layering. High level interfaces (like JobManagement) and low level interfaces (like Stream) are not distinguished, motivated, or put into context. E.g. why is there a need of a networkstream-level interface for a grid application? -> intentional, see use cases! -> DONE: ANDRE: CLEARIFY in intro - Example (P18): OO structured, not like the iface. (eg. i = dir.getNumEntries() instead of dir.getNumEntries(a) ? ) -> TODO ANDRE: cleanup, also session handle etc. - Example (P29): an exact copy of Ex. 1. -> check -> DONE ANDRE - Description - Streams (P62): close or destroyStream? maybe close, but not consistent. -> see Notes in Streams -> DONE ANDRE: use <constructor> or so. - check SIDL ==> not specified in SIDL, proposal: use "constructor" and "destructor" - General: sometimes bitwise OR-s are used for multiple options (Stream/ActivityType, P59), sometimes arrays of values (NameSpace/CopyFlags, P9). Why no convention? -> I agree, bitwise OR is simplier -> TODO ANDRE: check for languages, fix - Interface NSDir: what are the semantics of the copy method? of getName and getURL? -> ? -> DONE ANDRE: ask for clearification, sync with GFS Thorsten Sch"utt: - isFile -> isEntry? -> should we revive NSEntry (more reasons?)? -> TODO ANDRE: ask mailing list - SIDL docs on wiki! -> TODO TOM/JOHN - fix all examples in respect to session and context -> DONE ANDRE. Use a default session handle if none is specified -> examples are still valid :-D - flags -> TODO ANDRE: see above - Attribute -> AttributeSet -> TODO TOM: leave naming, but better describe idea Steven Newhouse: ---------------- - Inconsistent use of security! Its only in the stream section, nowhere else. "Not specifying a security system makes sense, but I think all of these methods should have a security token being passed into them or it should be an argument in the relevant object constructor." -> should be fixed by session and context -> DONE ANDRE - I assume the URL can support a number of protocols - http/https/file/gridftp. I see no way to register plugins in the interface (may be this is an 'internal' interface as opposed to a user funtion. Maybe there needs to be aet of interfaces to help the developers, e.g. register protocol plugins. -> plugin: implementation level -> url support: util class? any://? -> language specific -> need good error codes - State example language each time! Do examples in multiple languages! -> TODO: - How are you going to handle partial functionality implementations? Should _all_ operations support a NotImplementedException? Should there be a standard static method in each section to discover what is implemented? e.g. supported protocols & supported methods. -> methods need ALL to be implemented. -> protocols etc: hmmm... -> discussed at GGF, DONE -> was discussed at GGF... -> there WILL be partial implementation, but they are not SAGA conform then. -> necessary (session, context, ...) -> sufficient (one other subsys) -> CAN implement subsystems -> complete subsystems (partial compliant) -> TODO ANDRE - You have an 'LSF' schema for >>, >, < & <<. But in the file/directory area you use a series of attribute flags. I think you should carry these attribute flags forward into the job definition area - either way I think you should have one model not two! -> TODO ANDRE: use flags --------------------------------------------------------------------- - I. General Model. Language mapping. Standard attribute model. Security. Tasks II. Tier 1 Interfaces. III. Tier 2 and above Interfaces. -> II and III on same level! - What is to come in version 2? -> - steering and monitoring - possibly combining logical/physical files (read on logical files) - advert service - GridRPC - GridCPR - Task dependencies (simple work flows and batches) - extensions to existing classes - we have no explicit roadmap!! Others: - 'The URL Problem' -> should we approach it? - advert service -> easy: NS + Attribute + find on attribute! why not do it? - find in name spaces? -> why not, at least for names!? But over attributes? What about query language then? Andre: - list only on dir - why not on file? - go from getSize to real stat - namespace ops on files - why not on wildcards? - should callbacks be introduced for some things? Best on tasks, but also for streams (is factory!) Necessary for monitoring/streaming, so best introduce in general... - integrate GridRPC / GridCPR / JSDL? What else? - check if bulk operations can be done with task containers (explicit support needef for argument list generations?) - ACLs - Authorization (call outs?) - task = ftf.read (1, buffer, &n); task.run (); When are 'n' and 'buffer' available? I would say after wait. - default: define semantics, point out parts where semantics is up to the implementation GGF14: - who sends UC doc to editors? +-------------------------------------------------------------+

1 0

Re: comment on SAGA strawman doc.
by Andre Merzky 21 Jul '05

21 Jul '05

Dear Peter, Kalman, many thanks for your detailed comments! We will try to answer them all, but it will take us some time. After a first glance, they look very useful to me! I forward the mail to the SAGA list, for archiving and comments. Cheers, Andre. Quoting [Peter Kunszt] (May 31 2005): > Subject: comment on SAGA strawman doc. > Date: Tue, 31 May 2005 18:08:50 +0200 > From: "Peter Kunszt" <Peter.Kunszt(a)cern.ch> > To: "EGEE Middleware Engineering Management Team" <project-eu-egee-middleware-emt(a)cern.ch> > Cc: "EGEE Middleware Design Team" <project-eu-egee-middleware-design(a)cern.ch>, > "Kalman Kovari" <Kalman.Kovari(a)cern.ch>, > <andre(a)merzky.net> > > hi all > > following up on our task to be interoperable with > standards and to see how we fit into the large world > of GGF, we have looked at the strawman SAGA API doc . > http://wiki.cct.lsu.edu/saga/space/start/strawman-api.pdf > we focussed on the data management side, to see whether we > could implement these interfaces 'easily' or not. > > some of the comments are a bit sketchy.. glancing > at the SAGA mailinglist, some may have been discussed already. > > so we have only looked at the APIs relating to data management. > > the good news is that it seems that with some minor exceptions it would > be quite straightforward > to map the SAGA API to the existing Data Management APIs of the Fireman, > glite-io and the metadata catalog. it is not a surprise that the SAGA > API is much more sketchy and seems at times inconsistent and incomplete > compared with our > APIs since they don't have a real implementation behind them yet. > > the only 'issue' is the introduction of a stream-based network > API in SAGA. in our architecture / design we decided to go with the > managed async data transfer model (FTS), not with direct synchronous > streaming. the > reason is obvious: we need to manage the network resource efficiently. > > unfortunately the SAGA doc does not motivate the interfaces to an > extent where we would be able to deduce an 'architecture'... > we might also argue that some interfaces (like replica location, storage > space management, > lifetime management, quotas, etc) are missing... > > cheers > peter > > ps. kalman kovari [cc'd] did the actual work, i am just echoing and > commenting > his findings. i also cc'd andre merzky of the SAGA-WG. > > -------------- > > some detailed doc comments: > > - The interfaces in the document have some self-inconsistencies, > however, none of them is critical. > We expect these will be cleaned whenever someone starts to implement > them. > > - The examples are sometimes more misleading than useful... > > - The error handling and error semantics are not defined well enough. > This is essential > in order to assure proper interoperability. > > - There is a lack of structure in terms of interface layering. > High level interfaces (like JobManagement) and low level interfaces > (like > Stream) are not distinguished, motivated, or put into context. > E.g. why is there a need of a networkstream-level interface for a > grid application? > > - Example (P18): OO structured, not like the iface. (eg. i = > dir.getNumEntries() > instead of dir.getNumEntries(a) ? ) > > - Example (P29): an exact copy of Ex. 1. > > - Description - Streams (P62): close or destroyStream? maybe close, but > not consistent. > > - General: sometimes bitwise OR-s are used for multiple options > (Stream/ActivityType, P59), sometimes arrays of values > (NameSpace/CopyFlags, P9). Why no convention? > > - Description: Stream/read: Throws: IncorrectState if stream not in > "NotConnected" state. typo. > > - Description: Stream/write: Same as above. C&P. > > - Description: StreamServer/create,destroy: create/createStreamServer, > destroy/destroyStreamServer? > > - Interface NSDir: what are the semantics of the copy method? of > getName and getURL? > > ----------------- > > some details concerning implementation mappings. > below we mention the classes to be implemented, and their methods: > (an 'ok' is put to methods that are trivial to be implemented using the > existing APIs. probably just > name and parameter translations are required in that cases) > > > iface NSDir - except for copy(), everything can be implemented using the > fireman-catalog. > > changeDir: clientside manipulation+validity check in the catalog. > list: we only implement it with limits -- some default limits are > required. > readlink: ok. > exists: getlfnstat+check. > isDir: getlfnstat+check. > isFile: getlfnstat+check. > isLink: getlfnstat+check. > getNumEntries: we have no efficient implementation, possible with > readDir(), but slow. > copy: What do they mean saying "copy"? Needs clarification. > link: ok. > move: ok. > remove: unlink(). > makeDir: ok. > > getURL, getName: undocd. Needs to be clarified what they mean here, but > probably only clientside manipulation. > > iface Directory extends NSDir - internal scheme to implement directory > services. we already have a working catalog, > so we only need to make an interface for it. > getSize: fireman::getlfnstat. > openDir: clientside consturctor+check in the catalog if name is valid. > open: open/create a file. glite_io::open() and for creating some > fireman::create, too. > > iface File - implementable through glite_io. > read, write, seek: ok. > > iface LogicalDirectory extends NSDir - in the implementation scheme we > have no difference > between Directory and LogicalDirectory > > iface LogicalFile > addLocation, removeLocation, listLocations: addReplica, removeReplica, > listReplicas. ok. > replicate: addReplica > > iface Stream, StreamServer - low level, seemingly managing network > streams. > [see comments above] > > iface Attributes - usual key/value pairs, with support for > vectorAttributes. > > not looked at: > iface Task, TaskContainer > iface JobInfo, Job, JobService -- +-----------------------------------------------------------------+ | Andre Merzky | phon: +31 - 20 - 598 - 7759 | | Vrije Universiteit Amsterdam (VU) | fax : +31 - 20 - 598 - 7653 | | Dept. of Computer Science | mail: merzky(a)cs.vu.nl | | De Boelelaan 1083a | www: http://www.merzky.net | | 1081 HV Amsterdam, Netherlands | | +-----------------------------------------------------------------+

1 1

Agenda items for conference call ?
by Tom Goodale 20 Jul '05

20 Jul '05

Hi, the next conference call is scheduled for tomorrow (details to follow). Is there anything in particular we should add to the agenda beyond continuing through the issue list and looking at the action items from last time ? Cheers, Tom

1 2

Minutes from ByteIO session at GGF14
by neil p chue hong 12 Jul '05

12 Jul '05

Synopsis -------- We discussed the use cases we had received and the draft interface specification. We appealed again for more use cases and agreed changes and additions to the interface Both sets of minutes from our session are attached - many thanks to all who participated (inc the DFDL, GFS and SAGA groups) and special thanks to Andre for the second set of minutes. cheers, neil -- Neil P Chue Hong | T: [+44] (0)131 650 5957 Project Manager, EPCC | F: [+44] (0)131 650 6555 Rm 2409, JCMB, Mayfield Rd. | E: N.ChueHong(a)epcc.ed.ac.uk Edinburgh, EH9 3JZ, UK | W: http://www.epcc.ed.ac.uk "A film is like a battleground. It's love, hate, action, violence, death - in a word, emotion." - Sam Fuller

1 0

Re: SAGA question (fwd)
by Andre Merzky 07 Jul '05

07 Jul '05

Dear Mark, Thanks fro reading the API spec! :-) My 2cent worth of comments to your questions below. Quoting [Shantenu Jha] (Jul 07 2005): > > ---------- Forwarded message ---------- > Date: Fri, 10 Jun 2005 14:12:40 +0100 (BST) > From: Mark McKeown <zzalsmm3(a)nessie.mcc.ac.uk> > Subject: SAGA question > > I have a quick SAGA question about the strawman > API - page 41: > > ..... > JobService myjs = SomeJobServiceFactory (...); > Job myjob = new Job (); > > myjs.submitJob (jobdef, myjob); > > while ( something ) > { > JobState myjobstate; > myjob.getJobState (myjobstate); > > if ( myjobstate == Running ) > etc... > ...... > > > The two issues I am concerned with are latency and partial > failure (see "A Note on Distributed Computing", > http://research.sun.com/techrep/1994/abstract-29.html) > > ->Latency > > Since the job is running remotely is it sensible for a client > to make a decision based on if it is running - by the time > the client has got the status message the state of the job may > have changed. You have the same race condition locally - just because the times are shorter they are less likely to occur, but they can: you do a ps, and then a kill - and voila, the job is already gone. Since the kill (or whatever you do) will deliver a good error description ("Job does not exist", "Job is already stopped" etc), your application should be able to handle that situation gracefully (just as in the local case, where errno or shell return value give similar infos). > ->Partial Failure > What happens if there is a network failure when I do > > myjob.getJobState (myjobstate); > > how will this error by handled by the API? You will get a different, explicit error message ("could not contact resource/resource manager" or so) with the description of the problem, not just a failure. However, we need to define the possible error codes more specifically in the specification, you certainly have a valid point. We are working on that (Tom today volonteered to have a closer look at the error system). Thanks for your feedback, Cheers, Andre. > cheers > MArk -- +-----------------------------------------------------------------+ | Andre Merzky | phon: +31 - 20 - 598 - 7759 | | Vrije Universiteit Amsterdam (VU) | fax : +31 - 20 - 598 - 7653 | | Dept. of Computer Science | mail: merzky(a)cs.vu.nl | | De Boelelaan 1083a | www: http://www.merzky.net | | 1081 HV Amsterdam, Netherlands | | +-----------------------------------------------------------------+

1 0

Current issue list
by Tom Goodale 07 Jul '05

07 Jul '05

Peter Kunszt <Peter.Kunszt(a)cern.ch> Kalman Kovari <Kalman.Kovari(a)cern.ch> - The examples are sometimes more misleading than useful... -> why? - The error handling and error semantics are not defined well enough. This is essential in order to assure proper interoperability. -> compile list of error codes, check for consistency - There is a lack of structure in terms of interface layering. High level interfaces (like JobManagement) and low level interfaces (like Stream) are not distinguished, motivated, or put into context. E.g. why is there a need of a networkstream-level interface for a grid application? -> intentional, see use cases! - Example (P18): OO structured, not like the iface. (eg. i = dir.getNumEntries() instead of dir.getNumEntries(a) ? ) -> ? - Example (P29): an exact copy of Ex. 1. -> check - Description - Streams (P62): close or destroyStream? maybe close, but not consistent. -> see Notes in Streams - General: sometimes bitwise OR-s are used for multiple options (Stream/ActivityType, P59), sometimes arrays of values (NameSpace/CopyFlags, P9). Why no convention? -> I agree, bitwise OR is simplier - Interface NSDir: what are the semantics of the copy method? of getName and getURL? -> ? Thorsten Sch"utt: - isFile -> isEntry? -> should we revive NSEntry (more reasons?)? - SIDL docs on wiki! -> TODO - fix all examples in respect to session and context -> TODO - flags -> see above - Attribute -> AttributeSet -> leave naming, but better describe idea Steven Newhouse: - Inconsistent use of security! Its only in the stream section, nowhere else. "Not specifying a security system makes sense, but I think all of thes emethods should have a security token being passed into them or it should be an argument in the relevant object constructor." -> should be fixed by session and context - I assume the URL can support a number of protocols - http/https/file/gridftp. I see no way to register plugins in the interface (may be this is an 'internal' interface as opposed to a user funtion. Maybe there needs to be aet of interfaces to help the developers, e.g. register protocol plugins. -> plugin: implementation level -> url support: util class? any://? - State example language each time! Do examples in multiple languages! -> TODO - How are you going to handle partial functionality implementations? Should _all_ operations support a NotImplementedException ? Should there be a standard static method in each section to discover what is implemented? e.g. supported protocols & supported methods. -> methods need ALL to be implemented. -> protocols etc: hmmm... - You have an 'LSF' schema for >>, >, < & <<. But in the file/directory area you use a series of attribute flags. I think you should carry these attribute flags forward into the job definition area - either way I think you should have one model not two! -> Chris, Andre: resolve! - I. General Model. Language mapping. Standard attribute model. Security. Tasks II. Tier 1 Interfaces. III. Tier 2 and above Interfaces. -> II and III on same level! - What is to come in version 2? -> - steering and monitoring - possibly combining logical/physical files (read on logical files) - advert service - GridRPC - GridCPR - Task dependencies (simple work flows and batches) - extensions to existing classes - we have no explicit roadmap!! Others: - 'The URL Problem' -> should we approach it? - advert service -> easy: NS + Attribute + find on attribute! why not do it? - find in name spaces? -> why not, at least for names!? But over attributes? What about query language then? Andre: - list only on dir - why not on file? - go from getSize to real stat - namespace ops on files - why not on wildcards? - should callbacks be introduced for some things? Best on tasks, but also for streams (is factory!) Necessary for monitoring/streaming, so best introduce in general... - integrate GridRPC / GridCPR / JSDL? What else? - check if bulk operations can be done with task containers (explicit support needef for argument list generations?) - ACLs - Authorization (call outs?) - task = ftf.read (1, buffer, &n); task.run (); When are 'n' and 'buffer' available? I would say after wait. GGF14: - who sends UC doc to editors? +-------------------------------------------------------------+

2 2

SAGA Conference Call
by Andre Merzky 05 Jul '05

05 Jul '05

Hi all, as we agreed at GGF14, there will be a bi-weekly SAGA conference call starting at Thursday, July 7th, at Thu 5:00 PM (Amsterdam) Thu 4:00 PM (London) Thu 10:00 AM (Baton Rouge) Thu 8:00 AM (Los Angeles) Agenda: - followup from GGF14 - report on req. doc progress Number: 0844.888.8888 (UK) 01805.004.102 (Germany) +44.870.088.5706 (others) Code: 808044 Cheers, Andre. -- +-----------------------------------------------------------------+ | Andre Merzky | phon: +31 - 20 - 598 - 7759 | | Vrije Universiteit Amsterdam (VU) | fax : +31 - 20 - 598 - 7653 | | Dept. of Computer Science | mail: merzky(a)cs.vu.nl | | De Boelelaan 1083a | www: http://www.merzky.net | | 1081 HV Amsterdam, Netherlands | | +-----------------------------------------------------------------+

2 1

Two more use cases from LSU
by Andrei Hutanu 04 Jul '05

04 Jul '05

Hi! For your consideration, two late (hopefully not too late) use cases for SAGA from CCT, one from the UCoMS project, and one for interactive visualization. Andrei SAGA Use: ======================= Name of use case: UCoMS Project Contact (name and address): Zhou Lei (zlei(a)cct.lsu.edu) Andrei Hutanu (ahutanu(a)cct.lsu.edu) Authors (if different form contact) : Zhou Lei (zlei(a)cct.lsu.edu) 1. General Information: ----------------------- This section consists of check-boxes to provide some context in which to evaluate the use case. 1.1 Which best describes your organization: Industry [ ] Academic [x] Other [ ] Please specify: ................................... 1.2 Application area: Astronomy [ ] Particle physics [ ] Bio-informatics [ ] Environmental Sc. [ ] Image analysis [ ] Other [x] Please specify: Petroleum engineering applications 1.3 Which of the following apply to or best describe this use case Multiple selections are possible, please prioritize with numbers from 1 (low) to 5 (high): Database [ ] Remote steering [2] Visualization [4] Security [4] Resource discovery [5] Resource scheduling [5] Workflow [5] Data movement [5] High Throughput Computing [ ] High Performance Computing [5] Other [ ] Please specify: ................................... 1.4 Are you an: Application user [ ] Application developer [ ] System administrator [ ] Service developer [ ] Computer science researcher [x] Other [x] Please specify: Middleware developer 2. Introduction: ---------------- 2.1 Provide a paragraph introduction to your use case. Background to the project is another alternative. (E.g. 100 words). The UCoMS research aims are to develop and deploy a Ubiquitous Computing and Monitoring System (UCoMS) for oil/gas exploitation and management. Being a nationally unique research cluster in IT for energy, UCoMS addresses key research issues in the areas of wireless network systems, grid computing, and application software. It includes three cohesive and interrelated areas of projects which together enable the construction of a useful UCoMS prototype for discovery and management of energy resources. The UCoMS proof-of-concept prototype will be developed and deployed utilizing several existing (possibly decommissioned) well platforms in the Gulf of Mexico. The technical solutions will be generally applicable to sensor networks, wireless communications, and grid computing. These solutions will effectively facilitate drilling and operational data logging and processing, on-platform information distribution and displaying, infrastructure monitoring/intrusion detection, seismic processing and inversion, and management of complex surface facilities and pipelines. Decommissioned well platforms can be monitored and safeguarded using UCoMS, with a potential of fostering new industries as well in the future. 2.2 Is there a URL with more information about the project ? http://www.ucoms.org 3. Use Case to Motivate Functionality Within a Simple API: A general application in our use case includes: Computational resource brokering Data collection from largely-distributed data sources Task migration and farming Result attainment Visualization We need simple API for the following operations: stage executable, find data location, describe resources, visualization. A simple API will be the interface to resource brokering, data collection, task migration, and visualization on devices. All these processes are executed automatically. Also, we may need a simple API to manage workflow operations. 4. Customers: ------------- Describe customers of this use case and their needs. In particular, where and how the use case occurs "in nature" and for whom it occurs. E.g. max 40 words Energy companies, energy analysts, government agency 5. Involved Resources: ---------------------- 5.1 List all the resources needed: e.g. what hardware, data, software might be involved. Hardware involved in: a. Large-scale compute facilities b. Data storage system c. Visualization devices for good graphics support d. Wireless network and sensors network Software: a.GAT b.Globus toolkit c.Storage management, such as CCT Archive tool based on GAT or SRB. d.Petroleum engineering simulation software, such as UTChem, IPARS, VIP, and data processing, such as Seismic Unix and Delivery. e.Task farming using CACTUS (or Condor) f.Triana for workflow 5.2 Are these resources geographically distributed? Absolutely. All parts of the system, such as storage system, compute facilities, and visualization devices, are distributed. 5.3 How many resources are involved in the use case? E.g. how many remote tasks are executing at the same time? Usually, there are three kinds of resources involved in: compute facilities, sensor network, data storage system, and visualization devices. A large number of tasks could be executed simultaneously. 5.4 Describe your codes and tools: what sort of license is available, e.g. open or closed source license; what sort of third party tools and libraries do you use, and what is their availability; do you regularly work from source code, or use pre-compiled applications; what languages are your applications developed in (if relevant), e.g. Fortran, C, C++, Java, Perl, or Python. In the project, C/C++/Fortran are involved in compute-intensive data analysis. XML is used for data description. Java/Perl are used for system management and monitoring. OpenGL and VRML will be used for visualization and rendering. Some of the simulation software packages are open source, such as UTChem, while others require licenses, such as VIP. All middleware that will be used is open source. 5.5 What information sources do you require, e.g. certificate authorities, or registries. Because of spatial variability, GIS based information management is needed. Seismic data and well log are critical. GSI-based simple CA is used. Since security of data and results is crucial, certificates are important. 5.6 Do you use any resources other than traditional compute or data resources, e.g. telescopes, microscopes, medical imaging instruments. Yes, because we include IBW, BGM for sensor/wireless devices. 5.7 How often is your application used on the grid or grid-like systems? [x] Exclusively [ ] Often (say 50-50) [ ] Occasionally on the grid, but mostly stand-alone [ ] Not at all yet, but the plan is to. 6. Environment: --------------- Provide a description of the environment your scenario runs in, for example the languages used, the tool-sets used, and the user environments (e.g. shell, scripting language, or portal). a.C/C++/Fortran for reservoir and drilling simulation, for well logging and processing, and for seismic processing and inversion. b.Java/shell/perl/xhtml/portal for information display by browser. It should be web-service based. c.VRML and OpenGL for visualization d.All kinds of computational Grid middleware, such as GAT, Condor-G, Globus toolkit, Triana, and so on, are used. e.Development IDE may be eclipse for Java and C/C++. f.Commercial petroleum engineering simulation software. 7. How the resources are selected: ---------------------------------- 7.1 Which resources are selected by users, which are inherent in the application, and which are chosen by system administrators, or by other means? E.g. who is specifying the architecture and memory to run the remote tasks? Some resources, such as large-scale compute facilities and data storage system, are inherent in our project. Others, such as for system management, depend on applications. 7.2 How are the resources selected? E.g. by OS, by CPU power, by memory, don't care, by cost, frequency of availability of information, size of datasets? CPU power, memory, and size of datasets are critical. 7.3 Are the resource requirements dynamic or static? Basically, resource requirements are static. It is decided in advance what resources a particular task will require. 8. Security Considerations: --------------------------- 8.1 What things are sensitive in this scenario: executable code, data, computer hardware? I.e. at what level are security measures used to determine access, if any? Commercial level security. 8.2 Do you have any existing security framework, e.g. Kerberos 5, Unicore, GSI, SSH, smartcards? GSI and SSH 8.3 What are your security needs: authentication, authorization, message protection, data protection, anonymisation, audit trail, or others? Mostly, authentication is our first concern. We also need authorization, and data protection. 8.4 What are the most important issues which would simplify your security solution? Simple API, simple deployment, integration with commodity technologies. Simple API, deployment, and integration with commodity technologies. 9. Scalability: --------------- What are the things which are important to scalability and to what scale - compute resources, data, networks ? Compute resources, network, and data. 10. Performance Considerations: ------------------------------- Explain any relevant performance considerations of the use case. High performance for data processing, such as seismic processing & inversion, reservoir simulation, real-time drilling control, is very important, which requires high performance compute facilities and high-speed access to data storage. 11. Grid Technologies currently used: ------------------------------------- If you are currently using or developing this scenario, which grid technologies are you using or considering? Firstly, we use GAT over Globus toolkit, Condor, and other data management services. Secondly, some grid middleware, such as VDT, SBR, DRMAA, IBP, NWS/Ganglia, would like to use with GAT. 12. What Would You Like an API to Look Like? -------------------------------------------- Suggest some functions and their prototypes which you would like in an API which would support your scenario. Here is a scenario with a pseudo code: // for data collection char **sourceData = {“protocol://hostname:000/data1”, “protocol://hostname2:000/data2”, “protocol://hostname3:000/data” } DataLocation dl = provisionData(sourceData); JobDescription job = createJobInstance(hardwareRequirement, softwareRequirement, executable_Location, dl, output_location, errorno) // automatically resource brokering and data moving. SubmitJob(job); // Maybe there is job checkpointing and migration in background. // optional for visualization PostProcessForRender(output_location); Also, we maybe need workflow control using simple API. 13. References: --------------- List references for further reading. http://www.ucoms.org http://www.gridlab.org http://www.lgc.com SAGA Use Case Template: ======================= Name of use case: Interactive Visualization Services Contact (name and address): Andrei Hutanu <ahutanu(a)cct.lsu.edu> 1. General Information: ----------------------- This section consists of check-boxes to provide some context in which to evaluate the use case. 1.1 Which best describes your organization: Industry [ ] Academic [X] Other [ ] Please specify: ................................ 1.2 Application area: Astronomy [ ] Particle physics [ ] Bio-informatics [ ] Environmental Sc. [ ] Image analysis [ ] Other [ ] Please specify: Distributed visualization, Grid computing 1.3 Which of the following apply to or best describe this use case Multiple selections are possible, please prioritize with numbers from 1 (low) to 5 (high): Database [ ] Remote steering [4] Visualization [5] Security [1] Resource discovery [4] Resource scheduling [4] Workflow [ ] Data movement [3] High Throughput Computing [ ] High Performance Computing [ ] Other [ ] Please specify: ................................ 1.4 Are you an: Application user [ ] Application developer [ ] System administrator [ ] Service developer [ ] Computer science researcher [X] Other [ ] Please specify: ................................ 2. Introduction: ---------------- 2.1 Provide a paragraph introduction to your use case. Background to the project is another alternative. (E.g. 100 words). Interactive visualization of large datasets needs resources. The memory, CPU and GPU cycles, software licenses, display and interaction resources needed to manage the ever-growing data sizes are not available on every researcher's desktop. The advances in Grid computing technologies through projects like Globus, Condor, Unicore and Awaki to name just a few have made the sharing and aggregation of computational resources possible. Compute clusters connected by fast optical networks can be used together to solve larger and more complicated problems than the ones that can be solved by the largest supercomputer in the world. Unfortunately, the visualization component needed to analyse the generated output is lagging behind. Part of the problem is that there are many very complex grid tools that can be used to distribute a visualization application. Both the complexity and diversity of these tools are barriers in the way of getting them adopted by developers of visualization tools. Collaborative visualization involving any type of sharing between two or more users adds new requirements to the visualization of large data sets. 2.2 Is there a URL with more information about the project ? Currently only : http://www.cct.lsu.edu/Visualization/VizServiceDemo/ 3. Use Case to Motivate Functionality Within a Simple API: ---------------------------------------------------------- Provide a scenario description to explain customers' needs. E.g. "move a file from A to B," "start a job." Please include figures if possible. If your use case requires multiple components of functionality, please provide separate descriptions for each component, bullet points of 50 words per functionality are acceptable. * A simulation has produced a large dataset ( > 500 Gb), and stored it on a high-performance data archive on a remote machine A. We have two users, B and C using mid-range visualization hardware (dual-processor, high-end graphics card, etc..) large displays and a fast network connections (e.g. 10Gig). The users and the data are in three distinct locations, and a high-end visualization machines D is accessible for both of them. The users want to start and share a visualization session of the above mentioned data. A, B, C and D and are interconnected by optical networks. The proposed scenario looks as following : One of the users decides to start the visualization software on D The visualization software will need to spawn off a process on a machine E "near" the data archive. This will be a data service used to load select data of interest for the visualization session as requested by the users. On D (the machine where the visualization software is running), data flowing from A through E is transformed in images and video signal and using specialized hardware and software (to avoid read-back from the graphics card) the video signal is transformed in HD digital video streams. The users consume the video streams and share the interaction with the visualization service. The networks between A, B, C, D, and E have to be provisioned before or at the time the visualization job is started. The networks, compute and visualization resources are part of a complex resource description that is used to submit the visualization job. Simple APIs are required for the following operations: * Scheduling and submitting the visualization job on machines D and E and the networks interconnecting all the machines * Accessing the data from the data source A on the data service E * Interprocess communication between D and E and between D and B and C respectively. (asynchronous) * A very similar scenario is valid for one user only. User B would like to use the high-end visualization machine D to be able to interactively render more data than he's able to render on his local machine. Data service running on machine E is performing compute or memory extensive operations that cannot be performed on the local machine. 4. Customers: ------------- Describe customers of this use case and their needs. In particular, where and how the use case occurs "in nature" and for whom it occurs. E.g. max 40 words Users that want to collaboratively or individually visualize data so large that it cannot be interactively visualized on their local machines. 5. Involved Resources: ---------------------- 5.1 List all the resources needed: e.g. what hardware, data, software might be involved. Data source is a large capacity storage system, the data service runs potentially on multiple machines. In the simplest case, the data service serves as a cache for the data, in more complex scenarios uses CPU power to compute visualization data on-the-fly. Visualization service runs on a visualization machine with at least one graphics pipe (potentially multiple cards). 5.2 Are these resources geographically distributed? Yes, all the components are on separate machines interconnected by high-speed networks. 5.3 How many resources are involved in the use case? E.g. how many remote tasks are executing at the same time? At least two (data service and visualization service) but these can use multiple processors. 5.4 Describe your codes and tools: what sort of license is available, e.g. open or closed source license; what sort of third party tools and libraries do you use, and what is their availability; do you regularly work from source code, or use pre-compiled applications; what languages are your applications developed in (if relevant), e.g. Fortran, C, C++, Java, Perl, or Python. C/C++ visualization applications. Looking for general framework that can be applied to a wide range of applications. The framework will be open source, working from source code. Using a SOAP engine for inter-task communication. 5.5 What information sources do you require, e.g. certificate authorities, or registries. Need information about the resources. 5.6 Do you use any resources other than traditional compute or data resources, e.g. telescopes, microscopes, medical imaging instruments. Not currently. 5.7 How often is your application used on the grid or grid-like systems? [X] Exclusively [ ] Often (say 50-50) [ ] Occasionally on the grid, but mostly stand-alone [ ] Not at all yet, but the plan is to. 6. Environment: --------------- Provide a description of the environment your scenario runs in, for example the languages used, the tool-sets used, and the user environments (e.g. shell, scripting language, or portal). Shell. Portal interaction possible in limited. 7. How the resources are selected: ---------------------------------- 7.1 Which resources are selected by users, which are inherent in the application, and which are chosen by system administrators, or by other means? E.g. who is specifying the architecture and memory to run the remote tasks? The task of this service framework is to automatically select a set of resources that makes the solving of the task possible. If there are many choices, user selection is possible. 7.2 How are the resources selected? E.g. by OS, by CPU power, by memory, don't care, by cost, frequency of availability of information, size of datasets? Network connectivity, memory and CPU. 7.3 Are the resource requirements dynamic or static? Once the resources are selected they shouldn't need to be changed. 8. Security Considerations: --------------------------- 8.1 What things are sensitive in this scenario: executable code, data, computer hardware? I.e. at what level are security measures used to determine access, if any? Data might be sensitive, as well as software resources (licenses). Hardware resources are always sensitive. 8.2 Do you have any existing security framework, e.g. Kerberos 5, Unicore, GSI, SSH, smartcards? Using GSI in various components. 8.3 What are your security needs: authentication, authorisation, message protection, data protection, anonymisation, audit trail, or others? Need, authentication and authorization and audit trail for all the components. Data protection possibly needed. 8.4 What are the most important issues which would simplify your security solution? Simple API, simple deployment, integration with commodity technologies. User-friendliness. User should only input a password requested by the application when needed. An operation should not fail until it explicitly asks the user for the required credential information. 9. Scalability: --------------- What are the things which are important to scalability and to what scale - compute resources, data, networks ? Data size can scale indefinitely. Network capacity up to the current optical network capacities -- tens of Gbit/s. Compute resources -- up to 5 components in the entire scenario. 10. Performance Considerations: ------------------------------- Explain any relevant performance considerations of the use case. Asynchronous transfer is required for efficient bandwidth usage (need ability to pipeline multiple transport operations). Network transfer mechanism to scale up to the current fastest capacity networks. Should be able to choose more security vs. more performance. API for remote file access should not limit the performance of the storage system. 11. Grid Technologies currently used: ------------------------------------- If you are currently using or developing this scenario, which grid technologies are you using or considering? Web (Grid) services, grid job submission. 12. What Would You Like an API to Look Like? -------------------------------------------- Suggest some functions and their prototypes which you would like in an API which would support your scenario. For data transport: //starts send operation. callback is called at //the end of the operation AsyncResult beginSend (void* buf, int bufsize, AsyncResultCallback cb, void* userData); //releases resources, if called before callback blocks endSend (AsyncResult handle); AsyncResult beginReceive (AsyncResultCallback cb, void* userData); Block* endReceive (AsyncResult handle); void releaseBlock (Block* block); AsyncResult beginConnect (AsyncResultCallback cb, void* userData); void endConnect (AsyncResult handle); class AsyncResult { //cancel the operation void cancel (); } typedef struct Block{ const void* const buffer; size_t size; } Block; For job submission: No detailed API but should be able to specify something like: //Unspecified resource -- only requirements ResourceDescription A = ResourceDescription(10 CPU, 40 Gbyte RAM, X Gflops); ResourceDescription B = ResourceDescription(4 GraphicPipes, 4 Gbyte Video Ram) + A; //semi-specified resource : One of the machines that //holds a copy of my data file ResourceDescription C = ResourceDescription(LogicalFile("my_interesting_file")); //specified resource : the two users ResourceDescription D = ResourceDescription("client1.lsu.edu"); ResourceDescription E = ResourceDescription("client2.lsu.edu"); //hybrid resource .. some specified, some unspecified resources //that need to be connected by a 10Gbit/s network ResourceDescription F = ResourceDescription(Network(A, B, C, D, E, F, 10 Gbit/s)) + JobSubmit(myJob, F); 13. References: --------------- List references for further reading. http://wiki.cct.lsu.edu/wiki/space/Andrei+Hutanu/Prohaska_et_al_MardiGras20… http://www.cct.lsu.edu/Visualization/VizServiceDemo/ http://www.zib.de/visual/projects/gridlab/hdf5/ more to come..

2 1