June 2005 - saga-rg - lists.ogf.org

Re: comment on SAGA strawman doc.
by Andre Merzky 21 Jul '05

21 Jul '05

Dear Peter, Kalman, many thanks for your detailed comments! We will try to answer them all, but it will take us some time. After a first glance, they look very useful to me! I forward the mail to the SAGA list, for archiving and comments. Cheers, Andre. Quoting [Peter Kunszt] (May 31 2005): > Subject: comment on SAGA strawman doc. > Date: Tue, 31 May 2005 18:08:50 +0200 > From: "Peter Kunszt" <Peter.Kunszt(a)cern.ch> > To: "EGEE Middleware Engineering Management Team" <project-eu-egee-middleware-emt(a)cern.ch> > Cc: "EGEE Middleware Design Team" <project-eu-egee-middleware-design(a)cern.ch>, > "Kalman Kovari" <Kalman.Kovari(a)cern.ch>, > <andre(a)merzky.net> > > hi all > > following up on our task to be interoperable with > standards and to see how we fit into the large world > of GGF, we have looked at the strawman SAGA API doc . > http://wiki.cct.lsu.edu/saga/space/start/strawman-api.pdf > we focussed on the data management side, to see whether we > could implement these interfaces 'easily' or not. > > some of the comments are a bit sketchy.. glancing > at the SAGA mailinglist, some may have been discussed already. > > so we have only looked at the APIs relating to data management. > > the good news is that it seems that with some minor exceptions it would > be quite straightforward > to map the SAGA API to the existing Data Management APIs of the Fireman, > glite-io and the metadata catalog. it is not a surprise that the SAGA > API is much more sketchy and seems at times inconsistent and incomplete > compared with our > APIs since they don't have a real implementation behind them yet. > > the only 'issue' is the introduction of a stream-based network > API in SAGA. in our architecture / design we decided to go with the > managed async data transfer model (FTS), not with direct synchronous > streaming. the > reason is obvious: we need to manage the network resource efficiently. > > unfortunately the SAGA doc does not motivate the interfaces to an > extent where we would be able to deduce an 'architecture'... > we might also argue that some interfaces (like replica location, storage > space management, > lifetime management, quotas, etc) are missing... > > cheers > peter > > ps. kalman kovari [cc'd] did the actual work, i am just echoing and > commenting > his findings. i also cc'd andre merzky of the SAGA-WG. > > -------------- > > some detailed doc comments: > > - The interfaces in the document have some self-inconsistencies, > however, none of them is critical. > We expect these will be cleaned whenever someone starts to implement > them. > > - The examples are sometimes more misleading than useful... > > - The error handling and error semantics are not defined well enough. > This is essential > in order to assure proper interoperability. > > - There is a lack of structure in terms of interface layering. > High level interfaces (like JobManagement) and low level interfaces > (like > Stream) are not distinguished, motivated, or put into context. > E.g. why is there a need of a networkstream-level interface for a > grid application? > > - Example (P18): OO structured, not like the iface. (eg. i = > dir.getNumEntries() > instead of dir.getNumEntries(a) ? ) > > - Example (P29): an exact copy of Ex. 1. > > - Description - Streams (P62): close or destroyStream? maybe close, but > not consistent. > > - General: sometimes bitwise OR-s are used for multiple options > (Stream/ActivityType, P59), sometimes arrays of values > (NameSpace/CopyFlags, P9). Why no convention? > > - Description: Stream/read: Throws: IncorrectState if stream not in > "NotConnected" state. typo. > > - Description: Stream/write: Same as above. C&P. > > - Description: StreamServer/create,destroy: create/createStreamServer, > destroy/destroyStreamServer? > > - Interface NSDir: what are the semantics of the copy method? of > getName and getURL? > > ----------------- > > some details concerning implementation mappings. > below we mention the classes to be implemented, and their methods: > (an 'ok' is put to methods that are trivial to be implemented using the > existing APIs. probably just > name and parameter translations are required in that cases) > > > iface NSDir - except for copy(), everything can be implemented using the > fireman-catalog. > > changeDir: clientside manipulation+validity check in the catalog. > list: we only implement it with limits -- some default limits are > required. > readlink: ok. > exists: getlfnstat+check. > isDir: getlfnstat+check. > isFile: getlfnstat+check. > isLink: getlfnstat+check. > getNumEntries: we have no efficient implementation, possible with > readDir(), but slow. > copy: What do they mean saying "copy"? Needs clarification. > link: ok. > move: ok. > remove: unlink(). > makeDir: ok. > > getURL, getName: undocd. Needs to be clarified what they mean here, but > probably only clientside manipulation. > > iface Directory extends NSDir - internal scheme to implement directory > services. we already have a working catalog, > so we only need to make an interface for it. > getSize: fireman::getlfnstat. > openDir: clientside consturctor+check in the catalog if name is valid. > open: open/create a file. glite_io::open() and for creating some > fireman::create, too. > > iface File - implementable through glite_io. > read, write, seek: ok. > > iface LogicalDirectory extends NSDir - in the implementation scheme we > have no difference > between Directory and LogicalDirectory > > iface LogicalFile > addLocation, removeLocation, listLocations: addReplica, removeReplica, > listReplicas. ok. > replicate: addReplica > > iface Stream, StreamServer - low level, seemingly managing network > streams. > [see comments above] > > iface Attributes - usual key/value pairs, with support for > vectorAttributes. > > not looked at: > iface Task, TaskContainer > iface JobInfo, Job, JobService -- +-----------------------------------------------------------------+ | Andre Merzky | phon: +31 - 20 - 598 - 7759 | | Vrije Universiteit Amsterdam (VU) | fax : +31 - 20 - 598 - 7653 | | Dept. of Computer Science | mail: merzky(a)cs.vu.nl | | De Boelelaan 1083a | www: http://www.merzky.net | | 1081 HV Amsterdam, Netherlands | | +-----------------------------------------------------------------+

1 1

Two more use cases from LSU
by Andrei Hutanu 04 Jul '05

04 Jul '05

Hi! For your consideration, two late (hopefully not too late) use cases for SAGA from CCT, one from the UCoMS project, and one for interactive visualization. Andrei SAGA Use: ======================= Name of use case: UCoMS Project Contact (name and address): Zhou Lei (zlei(a)cct.lsu.edu) Andrei Hutanu (ahutanu(a)cct.lsu.edu) Authors (if different form contact) : Zhou Lei (zlei(a)cct.lsu.edu) 1. General Information: ----------------------- This section consists of check-boxes to provide some context in which to evaluate the use case. 1.1 Which best describes your organization: Industry [ ] Academic [x] Other [ ] Please specify: ................................... 1.2 Application area: Astronomy [ ] Particle physics [ ] Bio-informatics [ ] Environmental Sc. [ ] Image analysis [ ] Other [x] Please specify: Petroleum engineering applications 1.3 Which of the following apply to or best describe this use case Multiple selections are possible, please prioritize with numbers from 1 (low) to 5 (high): Database [ ] Remote steering [2] Visualization [4] Security [4] Resource discovery [5] Resource scheduling [5] Workflow [5] Data movement [5] High Throughput Computing [ ] High Performance Computing [5] Other [ ] Please specify: ................................... 1.4 Are you an: Application user [ ] Application developer [ ] System administrator [ ] Service developer [ ] Computer science researcher [x] Other [x] Please specify: Middleware developer 2. Introduction: ---------------- 2.1 Provide a paragraph introduction to your use case. Background to the project is another alternative. (E.g. 100 words). The UCoMS research aims are to develop and deploy a Ubiquitous Computing and Monitoring System (UCoMS) for oil/gas exploitation and management. Being a nationally unique research cluster in IT for energy, UCoMS addresses key research issues in the areas of wireless network systems, grid computing, and application software. It includes three cohesive and interrelated areas of projects which together enable the construction of a useful UCoMS prototype for discovery and management of energy resources. The UCoMS proof-of-concept prototype will be developed and deployed utilizing several existing (possibly decommissioned) well platforms in the Gulf of Mexico. The technical solutions will be generally applicable to sensor networks, wireless communications, and grid computing. These solutions will effectively facilitate drilling and operational data logging and processing, on-platform information distribution and displaying, infrastructure monitoring/intrusion detection, seismic processing and inversion, and management of complex surface facilities and pipelines. Decommissioned well platforms can be monitored and safeguarded using UCoMS, with a potential of fostering new industries as well in the future. 2.2 Is there a URL with more information about the project ? http://www.ucoms.org 3. Use Case to Motivate Functionality Within a Simple API: A general application in our use case includes: Computational resource brokering Data collection from largely-distributed data sources Task migration and farming Result attainment Visualization We need simple API for the following operations: stage executable, find data location, describe resources, visualization. A simple API will be the interface to resource brokering, data collection, task migration, and visualization on devices. All these processes are executed automatically. Also, we may need a simple API to manage workflow operations. 4. Customers: ------------- Describe customers of this use case and their needs. In particular, where and how the use case occurs "in nature" and for whom it occurs. E.g. max 40 words Energy companies, energy analysts, government agency 5. Involved Resources: ---------------------- 5.1 List all the resources needed: e.g. what hardware, data, software might be involved. Hardware involved in: a. Large-scale compute facilities b. Data storage system c. Visualization devices for good graphics support d. Wireless network and sensors network Software: a.GAT b.Globus toolkit c.Storage management, such as CCT Archive tool based on GAT or SRB. d.Petroleum engineering simulation software, such as UTChem, IPARS, VIP, and data processing, such as Seismic Unix and Delivery. e.Task farming using CACTUS (or Condor) f.Triana for workflow 5.2 Are these resources geographically distributed? Absolutely. All parts of the system, such as storage system, compute facilities, and visualization devices, are distributed. 5.3 How many resources are involved in the use case? E.g. how many remote tasks are executing at the same time? Usually, there are three kinds of resources involved in: compute facilities, sensor network, data storage system, and visualization devices. A large number of tasks could be executed simultaneously. 5.4 Describe your codes and tools: what sort of license is available, e.g. open or closed source license; what sort of third party tools and libraries do you use, and what is their availability; do you regularly work from source code, or use pre-compiled applications; what languages are your applications developed in (if relevant), e.g. Fortran, C, C++, Java, Perl, or Python. In the project, C/C++/Fortran are involved in compute-intensive data analysis. XML is used for data description. Java/Perl are used for system management and monitoring. OpenGL and VRML will be used for visualization and rendering. Some of the simulation software packages are open source, such as UTChem, while others require licenses, such as VIP. All middleware that will be used is open source. 5.5 What information sources do you require, e.g. certificate authorities, or registries. Because of spatial variability, GIS based information management is needed. Seismic data and well log are critical. GSI-based simple CA is used. Since security of data and results is crucial, certificates are important. 5.6 Do you use any resources other than traditional compute or data resources, e.g. telescopes, microscopes, medical imaging instruments. Yes, because we include IBW, BGM for sensor/wireless devices. 5.7 How often is your application used on the grid or grid-like systems? [x] Exclusively [ ] Often (say 50-50) [ ] Occasionally on the grid, but mostly stand-alone [ ] Not at all yet, but the plan is to. 6. Environment: --------------- Provide a description of the environment your scenario runs in, for example the languages used, the tool-sets used, and the user environments (e.g. shell, scripting language, or portal). a.C/C++/Fortran for reservoir and drilling simulation, for well logging and processing, and for seismic processing and inversion. b.Java/shell/perl/xhtml/portal for information display by browser. It should be web-service based. c.VRML and OpenGL for visualization d.All kinds of computational Grid middleware, such as GAT, Condor-G, Globus toolkit, Triana, and so on, are used. e.Development IDE may be eclipse for Java and C/C++. f.Commercial petroleum engineering simulation software. 7. How the resources are selected: ---------------------------------- 7.1 Which resources are selected by users, which are inherent in the application, and which are chosen by system administrators, or by other means? E.g. who is specifying the architecture and memory to run the remote tasks? Some resources, such as large-scale compute facilities and data storage system, are inherent in our project. Others, such as for system management, depend on applications. 7.2 How are the resources selected? E.g. by OS, by CPU power, by memory, don't care, by cost, frequency of availability of information, size of datasets? CPU power, memory, and size of datasets are critical. 7.3 Are the resource requirements dynamic or static? Basically, resource requirements are static. It is decided in advance what resources a particular task will require. 8. Security Considerations: --------------------------- 8.1 What things are sensitive in this scenario: executable code, data, computer hardware? I.e. at what level are security measures used to determine access, if any? Commercial level security. 8.2 Do you have any existing security framework, e.g. Kerberos 5, Unicore, GSI, SSH, smartcards? GSI and SSH 8.3 What are your security needs: authentication, authorization, message protection, data protection, anonymisation, audit trail, or others? Mostly, authentication is our first concern. We also need authorization, and data protection. 8.4 What are the most important issues which would simplify your security solution? Simple API, simple deployment, integration with commodity technologies. Simple API, deployment, and integration with commodity technologies. 9. Scalability: --------------- What are the things which are important to scalability and to what scale - compute resources, data, networks ? Compute resources, network, and data. 10. Performance Considerations: ------------------------------- Explain any relevant performance considerations of the use case. High performance for data processing, such as seismic processing & inversion, reservoir simulation, real-time drilling control, is very important, which requires high performance compute facilities and high-speed access to data storage. 11. Grid Technologies currently used: ------------------------------------- If you are currently using or developing this scenario, which grid technologies are you using or considering? Firstly, we use GAT over Globus toolkit, Condor, and other data management services. Secondly, some grid middleware, such as VDT, SBR, DRMAA, IBP, NWS/Ganglia, would like to use with GAT. 12. What Would You Like an API to Look Like? -------------------------------------------- Suggest some functions and their prototypes which you would like in an API which would support your scenario. Here is a scenario with a pseudo code: // for data collection char **sourceData = {“protocol://hostname:000/data1”, “protocol://hostname2:000/data2”, “protocol://hostname3:000/data” } DataLocation dl = provisionData(sourceData); JobDescription job = createJobInstance(hardwareRequirement, softwareRequirement, executable_Location, dl, output_location, errorno) // automatically resource brokering and data moving. SubmitJob(job); // Maybe there is job checkpointing and migration in background. // optional for visualization PostProcessForRender(output_location); Also, we maybe need workflow control using simple API. 13. References: --------------- List references for further reading. http://www.ucoms.org http://www.gridlab.org http://www.lgc.com SAGA Use Case Template: ======================= Name of use case: Interactive Visualization Services Contact (name and address): Andrei Hutanu <ahutanu(a)cct.lsu.edu> 1. General Information: ----------------------- This section consists of check-boxes to provide some context in which to evaluate the use case. 1.1 Which best describes your organization: Industry [ ] Academic [X] Other [ ] Please specify: ................................ 1.2 Application area: Astronomy [ ] Particle physics [ ] Bio-informatics [ ] Environmental Sc. [ ] Image analysis [ ] Other [ ] Please specify: Distributed visualization, Grid computing 1.3 Which of the following apply to or best describe this use case Multiple selections are possible, please prioritize with numbers from 1 (low) to 5 (high): Database [ ] Remote steering [4] Visualization [5] Security [1] Resource discovery [4] Resource scheduling [4] Workflow [ ] Data movement [3] High Throughput Computing [ ] High Performance Computing [ ] Other [ ] Please specify: ................................ 1.4 Are you an: Application user [ ] Application developer [ ] System administrator [ ] Service developer [ ] Computer science researcher [X] Other [ ] Please specify: ................................ 2. Introduction: ---------------- 2.1 Provide a paragraph introduction to your use case. Background to the project is another alternative. (E.g. 100 words). Interactive visualization of large datasets needs resources. The memory, CPU and GPU cycles, software licenses, display and interaction resources needed to manage the ever-growing data sizes are not available on every researcher's desktop. The advances in Grid computing technologies through projects like Globus, Condor, Unicore and Awaki to name just a few have made the sharing and aggregation of computational resources possible. Compute clusters connected by fast optical networks can be used together to solve larger and more complicated problems than the ones that can be solved by the largest supercomputer in the world. Unfortunately, the visualization component needed to analyse the generated output is lagging behind. Part of the problem is that there are many very complex grid tools that can be used to distribute a visualization application. Both the complexity and diversity of these tools are barriers in the way of getting them adopted by developers of visualization tools. Collaborative visualization involving any type of sharing between two or more users adds new requirements to the visualization of large data sets. 2.2 Is there a URL with more information about the project ? Currently only : http://www.cct.lsu.edu/Visualization/VizServiceDemo/ 3. Use Case to Motivate Functionality Within a Simple API: ---------------------------------------------------------- Provide a scenario description to explain customers' needs. E.g. "move a file from A to B," "start a job." Please include figures if possible. If your use case requires multiple components of functionality, please provide separate descriptions for each component, bullet points of 50 words per functionality are acceptable. * A simulation has produced a large dataset ( > 500 Gb), and stored it on a high-performance data archive on a remote machine A. We have two users, B and C using mid-range visualization hardware (dual-processor, high-end graphics card, etc..) large displays and a fast network connections (e.g. 10Gig). The users and the data are in three distinct locations, and a high-end visualization machines D is accessible for both of them. The users want to start and share a visualization session of the above mentioned data. A, B, C and D and are interconnected by optical networks. The proposed scenario looks as following : One of the users decides to start the visualization software on D The visualization software will need to spawn off a process on a machine E "near" the data archive. This will be a data service used to load select data of interest for the visualization session as requested by the users. On D (the machine where the visualization software is running), data flowing from A through E is transformed in images and video signal and using specialized hardware and software (to avoid read-back from the graphics card) the video signal is transformed in HD digital video streams. The users consume the video streams and share the interaction with the visualization service. The networks between A, B, C, D, and E have to be provisioned before or at the time the visualization job is started. The networks, compute and visualization resources are part of a complex resource description that is used to submit the visualization job. Simple APIs are required for the following operations: * Scheduling and submitting the visualization job on machines D and E and the networks interconnecting all the machines * Accessing the data from the data source A on the data service E * Interprocess communication between D and E and between D and B and C respectively. (asynchronous) * A very similar scenario is valid for one user only. User B would like to use the high-end visualization machine D to be able to interactively render more data than he's able to render on his local machine. Data service running on machine E is performing compute or memory extensive operations that cannot be performed on the local machine. 4. Customers: ------------- Describe customers of this use case and their needs. In particular, where and how the use case occurs "in nature" and for whom it occurs. E.g. max 40 words Users that want to collaboratively or individually visualize data so large that it cannot be interactively visualized on their local machines. 5. Involved Resources: ---------------------- 5.1 List all the resources needed: e.g. what hardware, data, software might be involved. Data source is a large capacity storage system, the data service runs potentially on multiple machines. In the simplest case, the data service serves as a cache for the data, in more complex scenarios uses CPU power to compute visualization data on-the-fly. Visualization service runs on a visualization machine with at least one graphics pipe (potentially multiple cards). 5.2 Are these resources geographically distributed? Yes, all the components are on separate machines interconnected by high-speed networks. 5.3 How many resources are involved in the use case? E.g. how many remote tasks are executing at the same time? At least two (data service and visualization service) but these can use multiple processors. 5.4 Describe your codes and tools: what sort of license is available, e.g. open or closed source license; what sort of third party tools and libraries do you use, and what is their availability; do you regularly work from source code, or use pre-compiled applications; what languages are your applications developed in (if relevant), e.g. Fortran, C, C++, Java, Perl, or Python. C/C++ visualization applications. Looking for general framework that can be applied to a wide range of applications. The framework will be open source, working from source code. Using a SOAP engine for inter-task communication. 5.5 What information sources do you require, e.g. certificate authorities, or registries. Need information about the resources. 5.6 Do you use any resources other than traditional compute or data resources, e.g. telescopes, microscopes, medical imaging instruments. Not currently. 5.7 How often is your application used on the grid or grid-like systems? [X] Exclusively [ ] Often (say 50-50) [ ] Occasionally on the grid, but mostly stand-alone [ ] Not at all yet, but the plan is to. 6. Environment: --------------- Provide a description of the environment your scenario runs in, for example the languages used, the tool-sets used, and the user environments (e.g. shell, scripting language, or portal). Shell. Portal interaction possible in limited. 7. How the resources are selected: ---------------------------------- 7.1 Which resources are selected by users, which are inherent in the application, and which are chosen by system administrators, or by other means? E.g. who is specifying the architecture and memory to run the remote tasks? The task of this service framework is to automatically select a set of resources that makes the solving of the task possible. If there are many choices, user selection is possible. 7.2 How are the resources selected? E.g. by OS, by CPU power, by memory, don't care, by cost, frequency of availability of information, size of datasets? Network connectivity, memory and CPU. 7.3 Are the resource requirements dynamic or static? Once the resources are selected they shouldn't need to be changed. 8. Security Considerations: --------------------------- 8.1 What things are sensitive in this scenario: executable code, data, computer hardware? I.e. at what level are security measures used to determine access, if any? Data might be sensitive, as well as software resources (licenses). Hardware resources are always sensitive. 8.2 Do you have any existing security framework, e.g. Kerberos 5, Unicore, GSI, SSH, smartcards? Using GSI in various components. 8.3 What are your security needs: authentication, authorisation, message protection, data protection, anonymisation, audit trail, or others? Need, authentication and authorization and audit trail for all the components. Data protection possibly needed. 8.4 What are the most important issues which would simplify your security solution? Simple API, simple deployment, integration with commodity technologies. User-friendliness. User should only input a password requested by the application when needed. An operation should not fail until it explicitly asks the user for the required credential information. 9. Scalability: --------------- What are the things which are important to scalability and to what scale - compute resources, data, networks ? Data size can scale indefinitely. Network capacity up to the current optical network capacities -- tens of Gbit/s. Compute resources -- up to 5 components in the entire scenario. 10. Performance Considerations: ------------------------------- Explain any relevant performance considerations of the use case. Asynchronous transfer is required for efficient bandwidth usage (need ability to pipeline multiple transport operations). Network transfer mechanism to scale up to the current fastest capacity networks. Should be able to choose more security vs. more performance. API for remote file access should not limit the performance of the storage system. 11. Grid Technologies currently used: ------------------------------------- If you are currently using or developing this scenario, which grid technologies are you using or considering? Web (Grid) services, grid job submission. 12. What Would You Like an API to Look Like? -------------------------------------------- Suggest some functions and their prototypes which you would like in an API which would support your scenario. For data transport: //starts send operation. callback is called at //the end of the operation AsyncResult beginSend (void* buf, int bufsize, AsyncResultCallback cb, void* userData); //releases resources, if called before callback blocks endSend (AsyncResult handle); AsyncResult beginReceive (AsyncResultCallback cb, void* userData); Block* endReceive (AsyncResult handle); void releaseBlock (Block* block); AsyncResult beginConnect (AsyncResultCallback cb, void* userData); void endConnect (AsyncResult handle); class AsyncResult { //cancel the operation void cancel (); } typedef struct Block{ const void* const buffer; size_t size; } Block; For job submission: No detailed API but should be able to specify something like: //Unspecified resource -- only requirements ResourceDescription A = ResourceDescription(10 CPU, 40 Gbyte RAM, X Gflops); ResourceDescription B = ResourceDescription(4 GraphicPipes, 4 Gbyte Video Ram) + A; //semi-specified resource : One of the machines that //holds a copy of my data file ResourceDescription C = ResourceDescription(LogicalFile("my_interesting_file")); //specified resource : the two users ResourceDescription D = ResourceDescription("client1.lsu.edu"); ResourceDescription E = ResourceDescription("client2.lsu.edu"); //hybrid resource .. some specified, some unspecified resources //that need to be connected by a 10Gbit/s network ResourceDescription F = ResourceDescription(Network(A, B, C, D, E, F, 10 Gbit/s)) + JobSubmit(myJob, F); 13. References: --------------- List references for further reading. http://wiki.cct.lsu.edu/wiki/space/Andrei+Hutanu/Prohaska_et_al_MardiGras20… http://www.cct.lsu.edu/Visualization/VizServiceDemo/ http://www.zib.de/visual/projects/gridlab/hdf5/ more to come..

2 1

New chairs
by Tom Goodale 30 Jun '05

30 Jun '05

Hi, we started the group with three chairs - myself, Stephen Pickles, and Keith Jackson - last year Stephen Pickles stepped down due to other commitments, and similarly last week Keith stepped down. We discussed this at a session in GGF yesterday, and are proposing to select Shantenu Jha and Andre Merzky to fill the vacant positions. If you have any objections or alternative candidates, please post them to the this list as soon as possible; failing objections I will inform the organisation of the new chairs during the week of the 11th of July. Cheers, Tom

1 0

Notes from the 3rd SAGA meeting at GGF14
by Thilo Kielmann 30 Jun '05

30 Jun '05

Simple API for Grid Applications (SAGA-RG) 3rd meeting at GGF14, thursday june 30, 7:30am - 9:00 am, room Regent II session chaired by Tom Goodale note taker: Thilo Kielmann Agenda: - introduction welcome and IPR issues note takers agenda bashing - requirements document structure, viewpoint - how do we proceed? - conclusion agenda bashing: Steven Newhouse: suggests to consider eventnotification (w.r.t. ws-notification) as an extension to the current SAGA API scope. The group will take this up as a "to do item". Requirements document Tom Goddale is going through the use cases. The group then identifies a structure for the document as follows: - introduction - target audience - where this fits into the SAGA procedure - use case analysis - subsection per use case . list functional areas . infrastructure requirements - requirements from other implementations . GAT, CoG, RealityGrid, EGGE g-lite, PyGlobus - infrastructure requirements/assumptions - functional areas . table/matrix . should be as explicit as possible - Summary . distill requirements into bullet-pointed list for final recommendations . split requirements into different phases of implementation based upon perceived importance/urgency. Conclusion review action items editors for requirements doc: Tom Goodale in august, Ed Seidel reviewers: Steven Newhouse, will bend arms to work on it send draft to mailinglist by sep 1st send individual bits to mailing list as soon as possible, even if piecemeal 8:30 meeting is closed

1 0

OGSA Byte-IO
by Jon MacLaren 24 Jun '05

24 Jun '05

Apologies if you know about this already, but there is a OGSA Byte-IO session on Wed. 11am next week at GGF. They are working on an OGSA style interface for the transfer of bulk data. It's going to be a small spec, just a couple of port types. They are aiming to have a draft recommendation document ready during the summer. One of the things they are providing is a POSIX-like interface for remote files. This might be of interest... Jon.

2 1

security info in streams...
by Andre Merzky 23 Jun '05

23 Jun '05

Hi All, John, the streams interface has a security info object, which allows to get information about the 'other' side of the stream. That overlaps somewhat with the context we intend to use to specify/query security information. Also, they are very similar: both are extending the attrributes interface, and specify a set of attributes to be supported. I'd like to propose to merge both, and use a read only context for security info in the stream interface. It makes sense to have dedicated GetXXX methods I think, for those attributes which are required. What do you think? Cheers. Andre. PS.: I know its summer, so you are probably all too lazy to dig through the doc for this :-) Here are the context and the security info: SecurityInfo: interface SecurityInfo extends-all SAGA.Attribute { /* These methods are shortcuts for typical * information that would be used to * make authorization decisions based on * connection information. However, the * the validity of the information is * dependent on the security model implementation. * Typically, the information is stored using * the SAGA.Attribute interface. The data * returned by the sample methods below are * also available via the Attribute interface. */ void getSourceUserName (out string name); void getSourceDN (out string DN); void getSourceHost (out string hostname); void getSourcePort (out int port); } Context: enum contextType { X509 = 0, SSH = 1, Kerberos = 2, UserPass = 3 }; interface Context extends-all SAGA.Attribute { constructor (in contextType type); getType (out contextType type); } -- +-----------------------------------------------------------------+ | Andre Merzky | phon: +31 - 20 - 598 - 7759 | | Vrije Universiteit Amsterdam (VU) | fax : +31 - 20 - 598 - 7653 | | Dept. of Computer Science | mail: merzky(a)cs.vu.nl | | De Boelelaan 1083a | www: http://www.merzky.net | | 1081 HV Amsterdam, Netherlands | | +-----------------------------------------------------------------+

4 6

Fwd (hupfeld@zib.de): Re: SAGA Strawman API Version 1.0
by Andre Merzky 20 Jun '05

20 Jun '05

Ha, finally someone really critical :-D Thanks Felix, comments are invited. Cheers, Andre. ----- Forwarded message from Felix Hupfeld <hupfeld(a)zib.de> ----- > I think there is a problem in the foundation of the SAGA approach. The > underlying assumptions seem to be that: > (a) prospective designers of Grid applications are unaware of distr. system > issues > (b) distr. system issues are not fundamental, but can be solved by > technical means > > on (a): Designers of Grid applications are presumably already writing > parallel applications on clusters or large HPC machines. They are well > aware that remote memory is not local memory and that they can't use DSM > just like local memory without serious performance issues. To name one > of the fundamental problems of writing distr. systems... > > on (b): There are fundamental tradeoffs to be considered when building > distributed systems (DS), like availability vs. consistency, > scalabillity vs. consistency, ... State of the art of CS is that they > can't be solved by technical means but are laws of the nature of DS. In > practice that means: if you design an API for a DS, where you _must_ > state which consistency guarentees you give, you assume certain > requirements of the user and force him into your choice of the tradeoff > spectrum. > > With assumptions (a) and (b), the designers of SAGA seem to persue an > approach, where the user has a very-POSIX like API. Although the spec. > ignores all DS issues, it seems to imply that the API gives the user a > single-copy view of the overall Grid, which makes the strongest choice > on consistency, but results in the worst possible scalability and > availability. > > To illustrate, I will elaborate on the File API, but the same arguments > can be made for namespaces and logical files, replicas, ... > > The spec. does not make any statements about the consistency guarantees > of the read and write operations. I assume that should imply that it > gives POSIX like read and write guarantees, resulting in bad scalability > and availability. The authors must explicitly state that. If the > consistency guarantees are left open and governed by the API > implementation, no applications can be written towards that API. You > can't write applications if you don't known when you will be able to > read the data you have written on local or remote instances. > > A solution to this problem would be the specification of required > consistency guarantees by the applications at open() time. This, > however, would require thorough research on conistency models and > application requirements, so I can't propose a few of hand. > > If SAGA choses to give single-system like guarantees, this must be > explicititely stated. All interfaces that deal with data are unusable > without a > specification of consistency guarantees. > > However, if you chose to be as POSIX like as possible and have > single-system semantics, why not directly use parts of the POSIX spec. > and strip it down where needed? ----- End forwarded message ----- -- +-----------------------------------------------------------------+ | Andre Merzky | phon: +31 - 20 - 598 - 7759 | | Vrije Universiteit Amsterdam (VU) | fax : +31 - 20 - 598 - 7653 | | Dept. of Computer Science | mail: merzky(a)cs.vu.nl | | De Boelelaan 1083a | www: http://www.merzky.net | | 1081 HV Amsterdam, Netherlands | | +-----------------------------------------------------------------+

4 7

proposal for extended file IO
by Andre Merzky 20 Jun '05

20 Jun '05

Hi again, consider following use case for remote IO. Given a large binary 2D field on a remote host, the client wans to access a 2D sub portion of that field. Dependend on the remote file layout, that requires usually more than one read operation, since the standard read (offset, length) is agnostic to the 2D layout. For more complex operations (subsampling, get a piece of a jpg file), the number of remote operations grow very fast. Latency then stringly discourages that type of remote IO. For that reason, I think that the remote file IO as specified by SAGA's Strawman as is will only be usable for a limited and trivial set of remote I/O use cases. There are three (basic) approaches: A) get the whole thing, and do ops locally Pro: - one remote op, - simple logic - remote side doesn't need to know about file structure - easily implementable on application level Con: - getting the header info of a 1GB data file comes with, well, some overhead ;-) B) clustering of calls: do many reads, but send them as a single request. Pro: - transparent to application - efficient Con: - need to know about dependencies of reads (a header read needed to determine size of field), or included explicite 'flushes' - need a protocol to support that - the remote side needs to support that C) data specific remote ops: send a high level command, and get exactly what you want. Pro: - most efficient Con: - need a protocol to support that - the remote side needs to support that _specific_ command The last approach (C) is what I have best experiences with. Also, that is what GridFTP as a common file access protocol supports via ERET/ESTO operations. I want to propose to include a C-like extension to the File API of the strawman, which basically maps well to GridFTP, but should also map to other implementations of C. That extension would look like: void lsEModes (out array<string,1> emodes ); void eWrite (in string emode, in string spec, in string buffer out long len_out ); void eRead (in string emode, in string spec, out string buffer, out long len_out ); - hooks for gridftp-like opaque ERET/ESTO features - spec: string for pattern as in GridFTP's ESTO/ERET - emode: string for ident. as in GridFTP's ESTO/ERET EMode: a specific remote I/O command supported lsEModes: list the EModes available in this implementation eRead/eWrite: read/write data according to the emode spec Example (in perl for brevity): my $file = SAGA::File new ("http://www.google.com/intl/en/images/logo.gif"); my @emodes = $file->lsEModes (); if ( grep (/^jpeg_block$/, @emodes) ) { my ($buff, $len) = file.eRead ("jpeg_block", "22x4+7+8"); } I would discourage support for B, since I do not know any protocoll supporting that approach efficiently, and also it needs approximately the same infrastructure setup as C. As A is easily implementable on application level, or within any SAGA implementation, there is no need for support on API level -- however, A is insufficient for all but some trivial cases. Comments welcome :-)) Cheers, Andre. -- +-----------------------------------------------------------------+ | Andre Merzky | phon: +31 - 20 - 598 - 7759 | | Vrije Universiteit Amsterdam (VU) | fax : +31 - 20 - 598 - 7653 | | Dept. of Computer Science | mail: merzky(a)cs.vu.nl | | De Boelelaan 1083a | www: http://www.merzky.net | | 1081 HV Amsterdam, Netherlands | | +-----------------------------------------------------------------+

6 35

scheduling at GGF14
by Andre Merzky 19 Jun '05

19 Jun '05

Hi all, following meetings have been scheduled for GGF14 (Thanks Tom and Shantenu :) SESSION 1: Date: Wednesday, Jun 29 Time: 7:30 a.m. - 9 a.m. Room: Regent 1 Agenda: - Usual formalities, Housekeeping and Introduction to group [10 min] - Raise the issue of "flipping a bit" - SAGA-RG to now WG? [10 min] - Move onto discussing the Strawman API and the way forward [70 min] Notable Conflicts: - Network Measurement - Configuration Description Deployment Lifecycle Mgmt SESSION 2: Date: Wednesday, Jun 29th Time: 2 p.m. - 3:30 p.m. Room: Regent 1 Agenda: - Continue discussion of API progress [90 minutes] Notable Conflicts: - GFS SESSION 3: Date: Thursday, Jun 30th Time: 7:30 a.m. - 9 a.m. Room: Regent 2 Agenda: - Open forum for creating and editing "Requirements Document" [90 min] Notable Conflicts: - JSDL :-( - Configuration Description Deployment Lifecycle Mgmt - GHPN Well, two times 7:30 - should give us a clean head, but we will also need some advertisement to convince people to come. And me ;-) Cheers, Andre -- +-----------------------------------------------------------------+ | Andre Merzky | phon: +31 - 20 - 598 - 7759 | | Vrije Universiteit Amsterdam (VU) | fax : +31 - 20 - 598 - 7653 | | Dept. of Computer Science | mail: merzky(a)cs.vu.nl | | De Boelelaan 1083a | www: http://www.merzky.net | | 1081 HV Amsterdam, Netherlands | | +-----------------------------------------------------------------+

1 0

planning for group meetings at GGF14?
by Thilo Kielmann 17 Jun '05

17 Jun '05

Hi SAGA folks, I may be missing something important, but do we already have a worked-out agenda for the session(s?) at GGF14? Our group is about to complete a 'strawman' implementation and we would like to 'show and get feedback'. Besides working on the obvious paper deliverables, will there be time for a talk with discussion about our SAGA implementation? See you in Chicago! Thilo -- Thilo Kielmann http://www.cs.vu.nl/~kielmann/

2 2