Hi everyone, <background> A little while ago, the WLCG VO ATLAS started investigating how to do storage accounting in a platform neutral fashion (i.e., not depending on SRM). Their idea was that a site would "somehow" ensure that their storage system always has a file containing the necessary information. The file would be stored in a standard location. ATLAS could then read this file using whichever protocol is easiest. The plan was for this file would be encoded as JSON. They also put together a rough description of how this file should look like. I suggested that they investigate GLUE 2 and the GLUE-JSON binding, rather than coming up with their own proprietary solution. </background> So, ATLAS are asking for some information: o people with whom they can discuss how to use GLUE2 and GLUE-JSON. o tools that parse GLUE-JSON. I guess this would be mostly from XSEDE guys. So, does anyone want to help them evaluate GLUE 2? Cheers, Paul.
Hi Paul, I had approached JP a few months ago with the idea of creating a schema in the Distributed Management Task Force (DMTF)'s new Redfish architecture specifically to accommodate GLUE data. This would have the following features and advantages: 1) Integrates with a standard already being built into hardware in data center server and storage by several vendors, 2) Would map well to the basic ideas and approach of GLUE 3) Provides existing implementation models using ODATA for organization and both JSON and XML representations for data exchange 4) DMTF would be willing to give us our own schema directory for Redfish for use with GLUE information. Redfish is being built into hardware from several major vendors now as a replacement for IPMI (for servers and similar equipment) with some features specifically interesting to GLUE for things like storage through the partnership between the Storage Networking Industry Association (SNIA) and DMTF. SNIA's version of Redfish is called Swordfish, and includes a lot of information that would be very good to use as input for gathering up information we normally care about representing at the data-center level with GLUE. I've thought of a name for this possible use of GLUE in the context of Redfish, which is of course "GLUEfish". The idea is to represent data center level information such as is typically gathered and exchanged in GLUE using the Redfish schema, thereby leveraging existing tools that could gather the information directly from current- and future-generation hardware. If you are interested in discussing this further, please let me know. Otherwise, here are some useful links: Redfish Developer Site: http://redfish.dmtf.org <http://redfish.dmtf.org/> Specifications: Redfish standard: http://www.dmtf.org/redfish <http://www.dmtf.org/redfish> OData: http://www.odata.org/documentation <http://www.odata.org/documentation> JSON-schema: http://www.json-schema.org <http://www.json-schema.org/> I'd love to talk about this more with the GLUE group in general. Alan
On Oct 12, 2016, at 5:25 PM, Paul Millar <paul.millar@desy.de> wrote:
Hi everyone,
<background>
A little while ago, the WLCG VO ATLAS started investigating how to do storage accounting in a platform neutral fashion (i.e., not depending on SRM).
Their idea was that a site would "somehow" ensure that their storage system always has a file containing the necessary information. The file would be stored in a standard location. ATLAS could then read this file using whichever protocol is easiest.
The plan was for this file would be encoded as JSON. They also put together a rough description of how this file should look like.
I suggested that they investigate GLUE 2 and the GLUE-JSON binding, rather than coming up with their own proprietary solution.
</background>
So, ATLAS are asking for some information:
o people with whom they can discuss how to use GLUE2 and GLUE-JSON.
o tools that parse GLUE-JSON. I guess this would be mostly from XSEDE guys.
So, does anyone want to help them evaluate GLUE 2?
Cheers,
Paul. _______________________________________________ glue-wg mailing list glue-wg@ogf.org https://www.ogf.org/mailman/listinfo/glue-wg
Alan, Having looked briefly at the DMTF Redfish architecture I notes their scope is to “meet(s) the expectations of Cloud and Web-based IT professionals for scalable platform management”. They describe their data model as "The Redfish model is built for managing systems.”. Looking at their data model and API they are focused on describing real and virtual hardware elements, and performing management operations on those elements. By contrast, GLUE2 is focused on user facing distributed resource metadata and interfaces to access those resources are out of scope (GLUE2 describes the interfaces, but they are implemented by other tools and specs). In short: - GLUE2 focuses on user facing information while Redfish focuses on infrastructure management information - GLUE2 focuses exclusively on discovery, while Redfish focuses on both discovery and performing actions on infrastructure While there is some modest overlap (both schemas have a concept of infrastructure location and administrative domains), they are trying to address different problems, IMO. Does anyone have other opinions on this? Thanks, JP
On Oct 12, 2016, at 6:02 PM, Alan Sill <kilohoku150@gmail.com> wrote:
Hi Paul,
I had approached JP a few months ago with the idea of creating a schema in the Distributed Management Task Force (DMTF)'s new Redfish architecture specifically to accommodate GLUE data. This would have the following features and advantages:
1) Integrates with a standard already being built into hardware in data center server and storage by several vendors, 2) Would map well to the basic ideas and approach of GLUE 3) Provides existing implementation models using ODATA for organization and both JSON and XML representations for data exchange 4) DMTF would be willing to give us our own schema directory for Redfish for use with GLUE information.
Redfish is being built into hardware from several major vendors now as a replacement for IPMI (for servers and similar equipment) with some features specifically interesting to GLUE for things like storage through the partnership between the Storage Networking Industry Association (SNIA) and DMTF. SNIA's version of Redfish is called Swordfish, and includes a lot of information that would be very good to use as input for gathering up information we normally care about representing at the data-center level with GLUE.
I've thought of a name for this possible use of GLUE in the context of Redfish, which is of course "GLUEfish". The idea is to represent data center level information such as is typically gathered and exchanged in GLUE using the Redfish schema, thereby leveraging existing tools that could gather the information directly from current- and future-generation hardware.
If you are interested in discussing this further, please let me know. Otherwise, here are some useful links:
Redfish Developer Site: http://redfish.dmtf.org <http://redfish.dmtf.org/>
Specifications:
Redfish standard: http://www.dmtf.org/redfish <http://www.dmtf.org/redfish> OData: http://www.odata.org/documentation <http://www.odata.org/documentation> JSON-schema: http://www.json-schema.org <http://www.json-schema.org/>
I'd love to talk about this more with the GLUE group in general.
Alan
On Oct 12, 2016, at 5:25 PM, Paul Millar <paul.millar@desy.de <mailto:paul.millar@desy.de>> wrote:
Hi everyone,
<background>
A little while ago, the WLCG VO ATLAS started investigating how to do storage accounting in a platform neutral fashion (i.e., not depending on SRM).
Their idea was that a site would "somehow" ensure that their storage system always has a file containing the necessary information. The file would be stored in a standard location. ATLAS could then read this file using whichever protocol is easiest.
The plan was for this file would be encoded as JSON. They also put together a rough description of how this file should look like.
I suggested that they investigate GLUE 2 and the GLUE-JSON binding, rather than coming up with their own proprietary solution.
</background>
So, ATLAS are asking for some information:
o people with whom they can discuss how to use GLUE2 and GLUE-JSON.
o tools that parse GLUE-JSON. I guess this would be mostly from XSEDE guys.
So, does anyone want to help them evaluate GLUE 2?
Cheers,
Paul. _______________________________________________ glue-wg mailing list glue-wg@ogf.org <mailto:glue-wg@ogf.org> https://www.ogf.org/mailman/listinfo/glue-wg
_______________________________________________ glue-wg mailing list glue-wg@ogf.org https://www.ogf.org/mailman/listinfo/glue-wg
Hi JP, I agree completely and may have missed stating my point clearly enough to say that. GLUE in general (correct me if I am wrong) helps describe data center capabilities and characteristics to each other -- i.e., is designed to facilitate describing and providing information appropriate to entire data centers or subsets of them at a large scale. Redfish applies to individual data center components, and is intended to replace IPMI for this purpose. IN addition, Swordfish aims to use the same schema structure and design to do the same sort of task in the storage domain. Many of the things that GLUE needs to gather up and aggregate, and present at a higher level, are explicitly provided for in Redfish and Swordfish at the individual component level for servers, storage units, etc. In addition, there are provisions in the Redfish and Swordfish designs to accommodate the concepts of collections as groupings of related capabilities. This is all designed to be schema-based (using OData and some related concepts) and programmable with modern RESTful tools using JSON or XML for data exchange. It seems to me that the capability to gather, parse, group, aggregate and present Swordfish and Redfish data using the concepts of GLUE would be a useful thing to have, especially because modern data center equipment is already being delivered with Redfish enabled and soon we will start to see Swordfish emerge for storage. Thus a layer to connect GLUE to these underlying sources of data, filling in from other sources where necessary, would seem to me to be a good idea. I am also interested in other opinions, so second your request. Hope this helps, Alan
On Oct 13, 2016, at 1:26 PM, Navarro, JP <navarro@mcs.anl.gov> wrote:
Alan,
Having looked briefly at the DMTF Redfish architecture I notes their scope is to “meet(s) the expectations of Cloud and Web-based IT professionals for scalable platform management”. They describe their data model as "The Redfish model is built for managing systems.”. Looking at their data model and API they are focused on describing real and virtual hardware elements, and performing management operations on those elements.
By contrast, GLUE2 is focused on user facing distributed resource metadata and interfaces to access those resources are out of scope (GLUE2 describes the interfaces, but they are implemented by other tools and specs).
In short: - GLUE2 focuses on user facing information while Redfish focuses on infrastructure management information - GLUE2 focuses exclusively on discovery, while Redfish focuses on both discovery and performing actions on infrastructure
While there is some modest overlap (both schemas have a concept of infrastructure location and administrative domains), they are trying to address different problems, IMO.
Does anyone have other opinions on this?
Thanks,
JP
On Oct 12, 2016, at 6:02 PM, Alan Sill <kilohoku150@gmail.com <mailto:kilohoku150@gmail.com>> wrote:
Hi Paul,
I had approached JP a few months ago with the idea of creating a schema in the Distributed Management Task Force (DMTF)'s new Redfish architecture specifically to accommodate GLUE data. This would have the following features and advantages:
1) Integrates with a standard already being built into hardware in data center server and storage by several vendors, 2) Would map well to the basic ideas and approach of GLUE 3) Provides existing implementation models using ODATA for organization and both JSON and XML representations for data exchange 4) DMTF would be willing to give us our own schema directory for Redfish for use with GLUE information.
Redfish is being built into hardware from several major vendors now as a replacement for IPMI (for servers and similar equipment) with some features specifically interesting to GLUE for things like storage through the partnership between the Storage Networking Industry Association (SNIA) and DMTF. SNIA's version of Redfish is called Swordfish, and includes a lot of information that would be very good to use as input for gathering up information we normally care about representing at the data-center level with GLUE.
I've thought of a name for this possible use of GLUE in the context of Redfish, which is of course "GLUEfish". The idea is to represent data center level information such as is typically gathered and exchanged in GLUE using the Redfish schema, thereby leveraging existing tools that could gather the information directly from current- and future-generation hardware.
If you are interested in discussing this further, please let me know. Otherwise, here are some useful links:
Redfish Developer Site: http://redfish.dmtf.org <http://redfish.dmtf.org/>
Specifications:
Redfish standard: http://www.dmtf.org/redfish <http://www.dmtf.org/redfish> OData: http://www.odata.org/documentation <http://www.odata.org/documentation> JSON-schema: http://www.json-schema.org <http://www.json-schema.org/>
I'd love to talk about this more with the GLUE group in general.
Alan
On Oct 12, 2016, at 5:25 PM, Paul Millar <paul.millar@desy.de <mailto:paul.millar@desy.de>> wrote:
Hi everyone,
<background>
A little while ago, the WLCG VO ATLAS started investigating how to do storage accounting in a platform neutral fashion (i.e., not depending on SRM).
Their idea was that a site would "somehow" ensure that their storage system always has a file containing the necessary information. The file would be stored in a standard location. ATLAS could then read this file using whichever protocol is easiest.
The plan was for this file would be encoded as JSON. They also put together a rough description of how this file should look like.
I suggested that they investigate GLUE 2 and the GLUE-JSON binding, rather than coming up with their own proprietary solution.
</background>
So, ATLAS are asking for some information:
o people with whom they can discuss how to use GLUE2 and GLUE-JSON.
o tools that parse GLUE-JSON. I guess this would be mostly from XSEDE guys.
So, does anyone want to help them evaluate GLUE 2?
Cheers,
Paul. _______________________________________________ glue-wg mailing list glue-wg@ogf.org <mailto:glue-wg@ogf.org> https://www.ogf.org/mailman/listinfo/glue-wg <https://www.ogf.org/mailman/listinfo/glue-wg>
_______________________________________________ glue-wg mailing list glue-wg@ogf.org <mailto:glue-wg@ogf.org> https://www.ogf.org/mailman/listinfo/glue-wg
Hi Alan, (see inline)
On Oct 13, 2016, at 3:28 PM, Alan Sill <kilohoku150@gmail.com> wrote:
Hi JP,
I agree completely and may have missed stating my point clearly enough to say that.
GLUE in general (correct me if I am wrong) helps describe data center capabilities and characteristics to each other -- i.e., is designed to facilitate describing and providing information appropriate to entire data centers or subsets of them at a large scale.
GLUE in general describes the user facing characteristics of distributed software, services, and batch execution environments. These elements may be on a single machine, on multiple machines within a center, or distributed across multiple data centers (when the GLUE data is aggregated).
Redfish applies to individual data center components, and is intended to replace IPMI for this purpose. IN addition, Swordfish aims to use the same schema structure and design to do the same sort of task in the storage domain.
Many of the things that GLUE needs to gather up and aggregate, and present at a higher level, are explicitly provided for in Redfish and Swordfish at the individual component level for servers, storage units, etc. In addition, there are provisions in the Redfish and Swordfish designs to accommodate the concepts of collections as groupings of related capabilities. This is all designed to be schema-based (using OData and some related concepts) and programmable with modern RESTful tools using JSON or XML for data exchange.
There may be a small subset of the hardware infrastructure characteristics (such as memory size) that do make it into user facing GLUE information. I can’t quantify it exactly without a detailed analysis, but I believe very few things that GLUE needs will be published using Redfish and Swordfish designs.
It seems to me that the capability to gather, parse, group, aggregate and present Swordfish and Redfish data using the concepts of GLUE would be a useful thing to have, especially because modern data center equipment is already being delivered with Redfish enabled and soon we will start to see Swordfish emerge for storage. Thus a layer to connect GLUE to these underlying sources of data, filling in from other sources where necessary, would seem to me to be a good idea.
Yeah, that is a possibility, to expand GLUE to include Redfish and Swordfish designed hardware infrastructure information. Are there any current or future GLUE users that want Redfish or Swordfish information? Is the DMTF looking for a higher level schema that they can plug Redfish and Swordfish into? JP
I am also interested in other opinions, so second your request.
Hope this helps, Alan
On Oct 13, 2016, at 1:26 PM, Navarro, JP <navarro@mcs.anl.gov <mailto:navarro@mcs.anl.gov>> wrote:
Alan,
Having looked briefly at the DMTF Redfish architecture I notes their scope is to “meet(s) the expectations of Cloud and Web-based IT professionals for scalable platform management”. They describe their data model as "The Redfish model is built for managing systems.”. Looking at their data model and API they are focused on describing real and virtual hardware elements, and performing management operations on those elements.
By contrast, GLUE2 is focused on user facing distributed resource metadata and interfaces to access those resources are out of scope (GLUE2 describes the interfaces, but they are implemented by other tools and specs).
In short: - GLUE2 focuses on user facing information while Redfish focuses on infrastructure management information - GLUE2 focuses exclusively on discovery, while Redfish focuses on both discovery and performing actions on infrastructure
While there is some modest overlap (both schemas have a concept of infrastructure location and administrative domains), they are trying to address different problems, IMO.
Does anyone have other opinions on this?
Thanks,
JP
On Oct 12, 2016, at 6:02 PM, Alan Sill <kilohoku150@gmail.com <mailto:kilohoku150@gmail.com>> wrote:
Hi Paul,
I had approached JP a few months ago with the idea of creating a schema in the Distributed Management Task Force (DMTF)'s new Redfish architecture specifically to accommodate GLUE data. This would have the following features and advantages:
1) Integrates with a standard already being built into hardware in data center server and storage by several vendors, 2) Would map well to the basic ideas and approach of GLUE 3) Provides existing implementation models using ODATA for organization and both JSON and XML representations for data exchange 4) DMTF would be willing to give us our own schema directory for Redfish for use with GLUE information.
Redfish is being built into hardware from several major vendors now as a replacement for IPMI (for servers and similar equipment) with some features specifically interesting to GLUE for things like storage through the partnership between the Storage Networking Industry Association (SNIA) and DMTF. SNIA's version of Redfish is called Swordfish, and includes a lot of information that would be very good to use as input for gathering up information we normally care about representing at the data-center level with GLUE.
I've thought of a name for this possible use of GLUE in the context of Redfish, which is of course "GLUEfish". The idea is to represent data center level information such as is typically gathered and exchanged in GLUE using the Redfish schema, thereby leveraging existing tools that could gather the information directly from current- and future-generation hardware.
If you are interested in discussing this further, please let me know. Otherwise, here are some useful links:
Redfish Developer Site: http://redfish.dmtf.org <http://redfish.dmtf.org/>
Specifications:
Redfish standard: http://www.dmtf.org/redfish <http://www.dmtf.org/redfish> OData: http://www.odata.org/documentation <http://www.odata.org/documentation> JSON-schema: http://www.json-schema.org <http://www.json-schema.org/>
I'd love to talk about this more with the GLUE group in general.
Alan
On Oct 12, 2016, at 5:25 PM, Paul Millar <paul.millar@desy.de <mailto:paul.millar@desy.de>> wrote:
Hi everyone,
<background>
A little while ago, the WLCG VO ATLAS started investigating how to do storage accounting in a platform neutral fashion (i.e., not depending on SRM).
Their idea was that a site would "somehow" ensure that their storage system always has a file containing the necessary information. The file would be stored in a standard location. ATLAS could then read this file using whichever protocol is easiest.
The plan was for this file would be encoded as JSON. They also put together a rough description of how this file should look like.
I suggested that they investigate GLUE 2 and the GLUE-JSON binding, rather than coming up with their own proprietary solution.
</background>
So, ATLAS are asking for some information:
o people with whom they can discuss how to use GLUE2 and GLUE-JSON.
o tools that parse GLUE-JSON. I guess this would be mostly from XSEDE guys.
So, does anyone want to help them evaluate GLUE 2?
Cheers,
Paul. _______________________________________________ glue-wg mailing list glue-wg@ogf.org <mailto:glue-wg@ogf.org> https://www.ogf.org/mailman/listinfo/glue-wg <https://www.ogf.org/mailman/listinfo/glue-wg>
_______________________________________________ glue-wg mailing list glue-wg@ogf.org <mailto:glue-wg@ogf.org> https://www.ogf.org/mailman/listinfo/glue-wg <https://www.ogf.org/mailman/listinfo/glue-wg>
Thanks for the reply and clarifications. On this:
On Oct 13, 2016, at 3:57 PM, Navarro, JP <navarro@mcs.anl.gov> wrote:
Is the DMTF looking for a higher level schema that they can plug Redfish and Swordfish into?
DMTF is willing to provide top-level schema space to other organizations (for example, OGF) in their schema repository for those organizations to use to maintain compatible schema of their own design that can be used with Redfish-compatible tools. (Swordfish, organized by the Storage Networking Industry Association, SNIA, is an example of such an organization that maintains its own schema. Of course, I am happy with whatever the group decides. Alan
Paul, We have a lot of distributed knowledge and experience in the glue-wg. How about if the join the glue-wg list and start the discussion there? If they could post a summary of their requirements and examples of their storage documents I’m sure several people will respond with ideas on whether they could leverage GLUE, or where GLUE may need to grow to satisfy their needs. JP
On Oct 12, 2016, at 5:25 PM, Paul Millar <paul.millar@desy.de> wrote:
Hi everyone,
<background>
A little while ago, the WLCG VO ATLAS started investigating how to do storage accounting in a platform neutral fashion (i.e., not depending on SRM).
Their idea was that a site would "somehow" ensure that their storage system always has a file containing the necessary information. The file would be stored in a standard location. ATLAS could then read this file using whichever protocol is easiest.
The plan was for this file would be encoded as JSON. They also put together a rough description of how this file should look like.
I suggested that they investigate GLUE 2 and the GLUE-JSON binding, rather than coming up with their own proprietary solution.
</background>
So, ATLAS are asking for some information:
o people with whom they can discuss how to use GLUE2 and GLUE-JSON.
o tools that parse GLUE-JSON. I guess this would be mostly from XSEDE guys.
So, does anyone want to help them evaluate GLUE 2?
Cheers,
Paul. _______________________________________________ glue-wg mailing list glue-wg@ogf.org https://www.ogf.org/mailman/listinfo/glue-wg
We have a lot of distributed knowledge and experience in the glue-wg. How about if the join the glue-wg list and start the discussion there? If they could post a summary of their requirements and examples of their storage documents I’m sure several people will respond with ideas on whether they could leverage GLUE, or where GLUE may need to grow to satisfy their needs. It would be good if they would do that but knowing ATLAS they will not want to be joined up to the list even if they could leave it later; I strongly suspect they will prefer a separate mailing list which is focused on their stuff. It would also be easier to resurrect it if
On 13/10/2016 19:03, Navarro, JP wrote: there are questions in the future. I agree with Paul, and as I have mentioned also in the GridPP meetings [*], it would be marvellous to not only not reinvent the wheel but also use a standard wheel that has already been invented. However, I think we will need to be pragmatic and work with them on their terms - CERN logins, CERN mailing lists, WLCG twiki if need be. I am also willing to contribute (as one of the people behind the CASTOR information provide at RAL where we still have a *cough* open ticket about implementing GLUE2...) Cheers --jens [*] http://storage.esc.rl.ac.uk/weekly/20160914-minutes.txt
If someone could post to the GLUE-WG list the coordinates to the ATLAS forum where they are discussing options the most active and interested GLUE-GW participants could join their discussion. JP
On Oct 14, 2016, at 6:35 AM, Jensen, Jens (STFC,RAL,SC) <jens.jensen@stfc.ac.uk> wrote:
We have a lot of distributed knowledge and experience in the glue-wg. How about if the join the glue-wg list and start the discussion there? If they could post a summary of their requirements and examples of their storage documents I’m sure several people will respond with ideas on whether they could leverage GLUE, or where GLUE may need to grow to satisfy their needs. It would be good if they would do that but knowing ATLAS they will not want to be joined up to the list even if they could leave it later; I strongly suspect they will prefer a separate mailing list which is focused on their stuff. It would also be easier to resurrect it if
On 13/10/2016 19:03, Navarro, JP wrote: there are questions in the future.
I agree with Paul, and as I have mentioned also in the GridPP meetings [*], it would be marvellous to not only not reinvent the wheel but also use a standard wheel that has already been invented.
However, I think we will need to be pragmatic and work with them on their terms - CERN logins, CERN mailing lists, WLCG twiki if need be. I am also willing to contribute (as one of the people behind the CASTOR information provide at RAL where we still have a *cough* open ticket about implementing GLUE2...)
Cheers --jens
[*] http://storage.esc.rl.ac.uk/weekly/20160914-minutes.txt
_______________________________________________ glue-wg mailing list glue-wg@ogf.org https://www.ogf.org/mailman/listinfo/glue-wg
On 14/10/2016 14:33, Navarro, JP wrote:
If someone could post to the GLUE-WG list the coordinates to the ATLAS forum where they are discussing options the most active and interested GLUE-GW participants could join their discussion. I agree - I will see if I can find out - I may have the information somewhere, or I certainly know whom to ask - or Paul may have his own contacts?
I guess they will all be at CHEP at the moment - apropos, it might be worth looking at the entries for CHEP [1] and WLCG workshop [2] to see if there is anything on information systems and/or accounting. I will ask people when they're back next week to report whether there was something on accounting. Cheers --jens [1] https://indico.cern.ch/event/505613/ [2] https://indico.cern.ch/event/555063/
Dear all, I'm currently at CHEP and I did a presentation about the new IS we are working in. You can check the slides here: https://indico.cern.ch/event/505613/contributions/2227932/ I'm not sure whether there was something about accounting, I don't think so, at least not for WLCG accounting. I also attended the WLCG workshop during the weekend. Accounting was discussed during the WLCG workshop on the collaboration board. Otherwise, neither accounting nor information systems were discussed during this workshop. There is a new Storage Task Force led by Oliver Keeble and at the pre-GDB of September they discussed storage accounting. The ATLAS person who has started all this should probably join this task force and align with it, since Alessandro di Girolamo, who is also an ATLAS person is leading this storage accounting effort and there are already many ideas around this. https://indico.cern.ch/event/394833/ That's the place to discuss this. Regards, Maria
-----Original Message----- From: glue-wg [mailto:glue-wg-bounces@ogf.org] On Behalf Of Jensen, Jens (STFC,RAL,SC) Sent: 14 October 2016 17:53 To: John-Paul Navarro <navarro@mcs.anl.gov> Cc: OGF GLUE Working Group <glue-wg@ogf.org> Subject: Re: [glue-wg] ATLAS, GLUE and JSON
On 14/10/2016 14:33, Navarro, JP wrote:
If someone could post to the GLUE-WG list the coordinates to the ATLAS forum where they are discussing options the most active and interested GLUE- GW participants could join their discussion. I agree - I will see if I can find out - I may have the information somewhere, or I certainly know whom to ask - or Paul may have his own contacts?
I guess they will all be at CHEP at the moment - apropos, it might be worth looking at the entries for CHEP [1] and WLCG workshop [2] to see if there is anything on information systems and/or accounting. I will ask people when they're back next week to report whether there was something on accounting.
Cheers --jens
[1] https://indico.cern.ch/event/505613/ [2] https://indico.cern.ch/event/555063/
_______________________________________________ glue-wg mailing list glue-wg@ogf.org https://www.ogf.org/mailman/listinfo/glue-wg
Many thanks, Maria. I am already on Oliver's mailing list, and that's a good way to take the discussion forward ; but I think we will need a new list for this specific discussion, unless there is one already that we can resurrect. (I imagine there may be questions on your CRIC, too?) Cheers --jens On 14/10/2016 16:59, Maria Alandes Pradillo wrote:
Dear all,
I'm currently at CHEP and I did a presentation about the new IS we are working in. You can check the slides here: https://indico.cern.ch/event/505613/contributions/2227932/
I'm not sure whether there was something about accounting, I don't think so, at least not for WLCG accounting.
I also attended the WLCG workshop during the weekend. Accounting was discussed during the WLCG workshop on the collaboration board. Otherwise, neither accounting nor information systems were discussed during this workshop.
There is a new Storage Task Force led by Oliver Keeble and at the pre-GDB of September they discussed storage accounting. The ATLAS person who has started all this should probably join this task force and align with it, since Alessandro di Girolamo, who is also an ATLAS person is leading this storage accounting effort and there are already many ideas around this. https://indico.cern.ch/event/394833/ That's the place to discuss this.
Regards, Maria
-----Original Message----- From: glue-wg [mailto:glue-wg-bounces@ogf.org] On Behalf Of Jensen, Jens (STFC,RAL,SC) Sent: 14 October 2016 17:53 To: John-Paul Navarro <navarro@mcs.anl.gov> Cc: OGF GLUE Working Group <glue-wg@ogf.org> Subject: Re: [glue-wg] ATLAS, GLUE and JSON
On 14/10/2016 14:33, Navarro, JP wrote:
If someone could post to the GLUE-WG list the coordinates to the ATLAS forum where they are discussing options the most active and interested GLUE- GW participants could join their discussion. I agree - I will see if I can find out - I may have the information somewhere, or I certainly know whom to ask - or Paul may have his own contacts?
I guess they will all be at CHEP at the moment - apropos, it might be worth looking at the entries for CHEP [1] and WLCG workshop [2] to see if there is anything on information systems and/or accounting. I will ask people when they're back next week to report whether there was something on accounting.
Cheers --jens
[1] https://indico.cern.ch/event/505613/ [2] https://indico.cern.ch/event/555063/
_______________________________________________ glue-wg mailing list glue-wg@ogf.org https://www.ogf.org/mailman/listinfo/glue-wg
Hi Jens,
I am already on Oliver's mailing list, and that's a good way to take the discussion forward ; but I think we will need a new list for this specific discussion, unless there is one already that we can resurrect.
I just wanted to say that Alessandro and Oliver have been working on this for a while now. It's up to the relevant parties to decide whether a new mailing list should be created or not. I just wanted to avoid a parallel discussion. I've forwarded this thread to Alessandro in any case.
(I imagine there may be questions on your CRIC, too?)
I'm not sure what you mean. I'm happy to answer any questions related to CRIC, however CRIC has been discussed for more than one year now in the Information System Evolution Task Force. There have been many meetings and presentations in different places within WLCG like GDB and MB. Everything is documented in the twiki: https://twiki.cern.ch/twiki/bin/view/EGEE/WLCGISEvolution The task force is open to everyone interested in the topic. There is also an egroup to follow the discussions. Regards, Maria
On 14/10/2016 16:59, Maria Alandes Pradillo wrote:
Dear all,
I'm currently at CHEP and I did a presentation about the new IS we are working in. You can check the slides here: https://indico.cern.ch/event/505613/contributions/2227932/
I'm not sure whether there was something about accounting, I don't think so, at least not for WLCG accounting.
I also attended the WLCG workshop during the weekend. Accounting was discussed during the WLCG workshop on the collaboration board. Otherwise, neither accounting nor information systems were discussed during this workshop.
There is a new Storage Task Force led by Oliver Keeble and at the pre-GDB of September they discussed storage accounting. The ATLAS person who has started all this should probably join this task force and align with it, since Alessandro di Girolamo, who is also an ATLAS person is leading this storage accounting effort and there are already many ideas around this. https://indico.cern.ch/event/394833/ That's the place to discuss this.
Regards, Maria
-----Original Message----- From: glue-wg [mailto:glue-wg-bounces@ogf.org] On Behalf Of Jensen, Jens (STFC,RAL,SC) Sent: 14 October 2016 17:53 To: John-Paul Navarro <navarro@mcs.anl.gov> Cc: OGF GLUE Working Group <glue-wg@ogf.org> Subject: Re: [glue-wg] ATLAS, GLUE and JSON
On 14/10/2016 14:33, Navarro, JP wrote:
If someone could post to the GLUE-WG list the coordinates to the ATLAS forum where they are discussing options the most active and interested GLUE- GW participants could join their discussion. I agree - I will see if I can find out - I may have the information somewhere, or I certainly know whom to ask - or Paul may have his own contacts?
I guess they will all be at CHEP at the moment - apropos, it might be worth looking at the entries for CHEP [1] and WLCG workshop [2] to see if there is anything on information systems and/or accounting. I will ask people when they're back next week to report whether there was something on accounting.
Cheers --jens
[1] https://indico.cern.ch/event/505613/ [2] https://indico.cern.ch/event/555063/
_______________________________________________ glue-wg mailing list glue-wg@ogf.org https://www.ogf.org/mailman/listinfo/glue-wg
On 14/10/2016 17:39, Maria Alandes Pradillo wrote:
(I imagine there may be questions on your CRIC, too?) I'm not sure what you mean. I'm happy to answer any questions related to CRIC, however CRIC has been discussed for more than one year now in the Information System Evolution Task Force. There have been many meetings and presentations in different places within WLCG like GDB and MB. Everything is documented in the twiki: https://twiki.cern.ch/twiki/bin/view/EGEE/WLCGISEvolution The task force is open to everyone interested in the topic. There is also an egroup to follow the discussions.
Sorry, I meant from the GLUE folks. If we are creating a new list/forum for the GLUE folks to talk to WLCG and/or ATLAS folks, it might be new to them. Or maybe they know all about it already. Thanks --jens
Hi Jens,
Sorry, I meant from the GLUE folks. If we are creating a new list/forum for the GLUE folks to talk to WLCG and/or ATLAS folks, it might be new to them. Or maybe they know all about it already.
Myself, Maarten and Stephen are members of the IS TF. As far as WLCG is concerned, we don't need to create any new list, I'm coordinating the TF and I'm in both GLUE and IS TF mailing lists. For ATLAS, what I understood from Alessandro, is that they need some advice on how to use GLUE 2 to describe some storage related attributes. IMHO the core problem here is not GLUE 2, it is how to address storage accounting. I think once they decide how to do this, using GLUE 2 should be pretty easy. At the same time, I agree with you it would be good to provide them already with some examples. I personally suggest all GLUE experts who want to follow this discussion, to join Oliver's TF and then discuss there how to proceed. Regards, Maria
Hi all, Thanks for the feedback so far. Just to clarify what's happening here: ATLAS has decided to do storage accounting by requiring sites/storage to have a JSON file written a well-known location. This is not under discussion. They have formulated a custom/proprietary JSON format that includes all the information they want to know. The proposed mechanism and JSON file format has been presented in the September pre-GDB: https://indico.cern.ch/event/394833/#3-resource-reporting-proposal I suggested they consider GLUE2-JSON instead. To do consider GLUE2-JSON, they need: 1. some people with which they can discuss any issues, 2. example JSON files that provide complete examples. I suggest we tackle 1. first and look at 2. as an outcome of the people in 1. So, simple question to everyone in this group: Who wishes to help ATLAS evaluate GLUE2-JSON? Cheers, Paul.
Paul, As you know, XSEDE has a partial implementation of GLUE2-JSON in production. Our source code is available in GIT at: https://github.com/ericblau/ipf-xsede/ <https://github.com/ericblau/ipf-xsede/> Step 1, IMO, is for ATLAS to look at the GLUE2 data model and figure out how the information in their JSON file maps on to the GLUE2 reference model. Many GLUE2-WG members can help with that if they can’t do it on their own. Step 2 would then be to look at existing implementations, such as XSEDE’s, and decide whether to extend that implementation with the information they want, or to develop something new. XSEDE is available to discuss what it would take to add specific types of information if they can identify what those are. Perhaps they could share documentation and examples on those files for others to help with the above steps? JP
On Oct 14, 2016, at 1:23 PM, Paul Millar <paul.millar@desy.de> wrote:
Hi all,
Thanks for the feedback so far.
Just to clarify what's happening here: ATLAS has decided to do storage accounting by requiring sites/storage to have a JSON file written a well-known location. This is not under discussion.
They have formulated a custom/proprietary JSON format that includes all the information they want to know.
The proposed mechanism and JSON file format has been presented in the September pre-GDB:
https://indico.cern.ch/event/394833/#3-resource-reporting-proposal
I suggested they consider GLUE2-JSON instead. To do consider GLUE2-JSON, they need:
1. some people with which they can discuss any issues,
2. example JSON files that provide complete examples.
I suggest we tackle 1. first and look at 2. as an outcome of the people in 1.
So, simple question to everyone in this group:
Who wishes to help ATLAS evaluate GLUE2-JSON?
Cheers,
Paul. _______________________________________________ glue-wg mailing list glue-wg@ogf.org https://www.ogf.org/mailman/listinfo/glue-wg
Hi JP, On 14/10/16 20:40, Navarro, JP wrote:
As you know, XSEDE has a partial implementation of GLUE2-JSON in production. Our source code is available in GIT at: https://github.com/ericblau/ipf-xsede/
Great, thanks for the link.
Step 1, IMO, is for ATLAS to look at the GLUE2 data model and figure out how the information in their JSON file maps on to the GLUE2 reference model. Many GLUE2-WG members can help with that if they can’t do it on their own.
My reading of the situation is that they don't want to read all of GLUE2 and GLUE2-JSON binding documents and try to understand it all to evaluate whether GLUE2-JSON makes sense for them. Instead, I believe they want to use us as consultants; in particular, they want some concrete GLUE2-JSON examples to look at. Something like a template they can use. The pre-GDB link includes some examples in their proprietary format: https://indico.cern.ch/event/394833/#3-resource-reporting-proposal Can we describe this information in GLUE2-JSON?
Step 2 would then be to look at existing implementations, such as XSEDE’s, and decide whether to extend that implementation with the information they want, or to develop something new. XSEDE is available to discuss what it would take to add specific types of information if they can identify what those are.
Yes, I explained (or tried to explain) that XSEDE's existing code-base might or might not be useful for them.
Perhaps they could share documentation and examples on those files for others to help with the above steps?
Examples of their proprietary format are available from within Alessandro's Pre-GDB presentation. Cheers, Paul.
Paul, XSEDE’s implementation publishes these GLUE2 Entities: 'ApplicationEnvironment’, 'ApplicationHandle’, 'AbstractService’, 'Endpoint', 'ComputingManager’, 'ComputingShare’, ‘ComputingActivity', 'ExecutionEnvironment’, ‘Location' The only entities with storage relate information are AbstractService and Endpoint. XSEDE’s AbstractService and Endpoint information can be viewed here: https://info1.dyn.xsede.org:443/wh1/glue2-db-api/v1/abstractservice/ https://info1.dyn.xsede.org:443/wh1/glue2-db-api/v1/endpoint/ <https://info1.dyn.xsede.org/wh1/glue2-db-api/v1/endpoint/> Within this framework they could publish their storage dump (capacity_id) entries as abstract services with each of the paths as endpoints. Attributes that XSEDE currently doesn’t support could be added if they exist in the GLUE2 model, and if not included in Extensions as specified by GLUE2. JP
On Oct 14, 2016, at 2:16 PM, Paul Millar <paul.millar@desy.de> wrote:
My reading of the situation is that they don't want to read all of GLUE2 and GLUE2-JSON binding documents and try to understand it all to evaluate whether GLUE2-JSON makes sense for them.
Instead, I believe they want to use us as consultants; in particular, they want some concrete GLUE2-JSON examples to look at. Something like a template they can use.
The pre-GDB link includes some examples in their proprietary format:
https://indico.cern.ch/event/394833/#3-resource-reporting-proposal
Can we describe this information in GLUE2-JSON?
Step 2 would then be to look at existing implementations, such as XSEDE’s, and decide whether to extend that implementation with the information they want, or to develop something new. XSEDE is available to discuss what it would take to add specific types of information if they can identify what those are.
Yes, I explained (or tried to explain) that XSEDE's existing code-base might or might not be useful for them.
Perhaps they could share documentation and examples on those files for others to help with the above steps?
Examples of their proprietary format are available from within Alessandro's Pre-GDB presentation.
Cheers,
Paul.
glue-wg [mailto:glue-wg-bounces@ogf.org] On Behalf Of Navarro, JP said:
Within this framework they could publish their storage dump (capacity_id) entries as abstract services with each of the paths as endpoints.
This information would use StorageShareCapacity, and perhaps the parent StorageShare. At a quick look the information is generally similar, although some things, e.g. number of files, would need to be OtherInfo/Extension. Stephen
My impression looking at the sample JSON documents was also that the information ATLAS wants to publish easily maps to GLUE2. XSEDE is willing to assist in enhancing our open source implementation with these new storage entities by: - giving a teleconference presentation developers and system administrators on the “ipf-xsede” package - responding to e-mail from other developers who are enhancing it - conducting independent testing of contributions/enhancements - helping produce new releases and packages We are also willing to rename the product from “ipf-xsede” to “ipf” (which stands for "information publishing framework") or ipf-<whatever> if someone can think of a better <whatever>. Regards, JP
On Oct 15, 2016, at 3:17 AM, stephen.burke@stfc.ac.uk wrote:
glue-wg [mailto:glue-wg-bounces@ogf.org] On Behalf Of Navarro, JP said:
Within this framework they could publish their storage dump (capacity_id) entries as abstract services with each of the paths as endpoints.
This information would use StorageShareCapacity, and perhaps the parent StorageShare. At a quick look the information is generally similar, although some things, e.g. number of files, would need to be OtherInfo/Extension.
Stephen
Hi JP, On 17/10/16 17:36, Navarro, JP wrote:
My impression looking at the sample JSON documents was also that the information ATLAS wants to publish easily maps to GLUE2.
That was my impression, too.
XSEDE is willing to assist in enhancing our open source implementation with these new storage entities by:
- giving a teleconference presentation developers and system administrators on the “ipf-xsede” package
- responding to e-mail from other developers who are enhancing it
- conducting independent testing of contributions/enhancements
- helping produce new releases and packages
We are also willing to rename the product from “ipf-xsede” to “ipf” (which stands for "information publishing framework") or ipf-<whatever> if someone can think of a better <whatever>.
That's certainly a generous offer. I'll relay this back to the ATLAS team. Should there be a separate mailing list to continue this discussion, or should that take place on this list <glue-wg@ogf.org> ? Cheers, Paul.
On Oct 24, 2016, at 9:45 AM, Paul Millar <paul.millar@desy.de> wrote:
Should there be a separate mailing list to continue this discussion, or should that take place on this list <glue-wg@ogf.org <mailto:glue-wg@ogf.org>> ?
I’m flexible. It’s really up to the ATLAS folks and whether they are willing to join glue-wg (at least temporarily for this discussion) or do they have a different list they want GLUE folks to join. JP
As I understood Maria, enough people (=3?) from ATLAS-slash-CERN are on the GLUE list to continue the discussion on the GLUE list. However, this of course assumes that (1) the rest of the GLUE community are not unwilling to use the list for this purpose, and (2) that everyone else who wishes to contribute but is not on GLUE is willing to join the GLUE list, at least temporarily. For my much devalued twopenny's worth, I am happy to (a) contribute and (b) use the GLUE list. I have also advertised it to the rest of the GridPP storage group. In other words - "GridPP is willin'" (with a wink) ________________________________ From: glue-wg [glue-wg-bounces@ogf.org] on behalf of Navarro, JP [navarro@mcs.anl.gov] Sent: 24 October 2016 16:27 To: Paul Millar Cc: OGF GLUE Working Group Subject: Re: [glue-wg] ATLAS, GLUE and JSON On Oct 24, 2016, at 9:45 AM, Paul Millar <paul.millar@desy.de<mailto:paul.millar@desy.de>> wrote: Should there be a separate mailing list to continue this discussion, or should that take place on this list <glue-wg@ogf.org<mailto:glue-wg@ogf.org>> ? I’m flexible. It’s really up to the ATLAS folks and whether they are willing to join glue-wg (at least temporarily for this discussion) or do they have a different list they want GLUE folks to join. JP
hi Paul, On 2016.10.14. 20:23, Paul Millar wrote:
I suggested they consider GLUE2-JSON instead. To do consider GLUE2-JSON, they need:
1. some people with which they can discuss any issues,
2. example JSON files that provide complete examples.
I suggest we tackle 1. first and look at 2. as an outcome of the people in 1.
So, simple question to everyone in this group:
Who wishes to help ATLAS evaluate GLUE2-JSON?
You can count on me. Balazs Konya ps: i already gave some feedback to ATLAS wrt how they could make their storage record more glue compliant.
participants (8)
-
Alan Sill -
Balazs Konya -
jens.jensen＠stfc.ac.uk -
Jensen, Jens (STFC,RAL,SC)
-
Maria Alandes Pradillo -
Navarro, JP -
Paul Millar -
stephen.burke＠stfc.ac.uk