Hi all, Does anyone know if OSG are planning to drop support for publishing WLCG information via GLUE? I opened a GGUS ticket describing how some site-level information was wrongly formatted and the ticket was closed as WONT_FIX with the comment that "OSG is deprecating the BDII". Cheers, Paul.
Hi Paul,
Does anyone know if OSG are planning to drop support for publishing WLCG information via GLUE?
They will not be able to do such a thing very soon AFAIK, as ATLAS, CMS and the WLCG project still have dependencies on the classic info system, even if only to bootstrap various private or derived info systems. OTOH, OSG have managed to hold off GLUE-2 because none of those stakeholders made a strong case for it so far...
I opened a GGUS ticket describing how some site-level information was wrongly formatted and the ticket was closed as WONT_FIX with the comment that "OSG is deprecating the BDII".
Even if it were true for the long run, it does not mean they can get away with publishing junk already today. However, to get some problem fixed, it may need to be seen as an issue for one of those same stakeholders...
Dear all, I will try to clarify with OSG why they close the GGUS ticket with that answer. I agree with Maarten that either they have a proper plan to stop publishing resources in the BDII and announce it officially and make sure it´s not a problem for their users, or otherwise they should publish proper values. Regards, Maria
-----Original Message----- From: glue-wg-bounces@ogf.org [mailto:glue-wg-bounces@ogf.org] On Behalf Of Maarten.Litmaath@cern.ch Sent: 08 May 2015 02:03 To: Paul Millar Cc: glue-wg@ogf.org Subject: Re: [glue-wg] OSG dropping GLUE?
Hi Paul,
Does anyone know if OSG are planning to drop support for publishing WLCG information via GLUE?
They will not be able to do such a thing very soon AFAIK, as ATLAS, CMS and the WLCG project still have dependencies on the classic info system, even if only to bootstrap various private or derived info systems.
OTOH, OSG have managed to hold off GLUE-2 because none of those stakeholders made a strong case for it so far...
I opened a GGUS ticket describing how some site-level information was wrongly formatted and the ticket was closed as WONT_FIX with the comment that "OSG is deprecating the BDII".
Even if it were true for the long run, it does not mean they can get away with publishing junk already today. However, to get some problem fixed, it may need to be seen as an issue for one of those same stakeholders...
_______________________________________________ glue-wg mailing list glue-wg@ogf.org https://www.ogf.org/mailman/listinfo/glue-wg
Hi Maria, Welcome back! On 08/05/15 09:20, Maria Alandes Pradillo wrote:
I will try to clarify with OSG why they close the GGUS ticket with that answer. I agree with Maarten that either they have a proper plan to stop publishing resources in the BDII and announce it officially and make sure it´s not a problem for their users, or otherwise they should publish proper values.
Thanks. I don't think I mentioned the ticket number in my earlier email. If it helps, here's the actual ticket: https://ggus.eu/index.php?mode=ticket_info&ticket_id=113579 Cheers, Paul.
Yes, thanks a lot! I was going to look for the ticket in GGUS :-)
-----Original Message----- From: Paul Millar [mailto:paul.millar@desy.de] Sent: 08 May 2015 10:16 To: Maria Alandes Pradillo; Maarten Litmaath Cc: glue-wg@ogf.org Subject: Re: [glue-wg] OSG dropping GLUE?
Hi Maria,
Welcome back!
On 08/05/15 09:20, Maria Alandes Pradillo wrote:
I will try to clarify with OSG why they close the GGUS ticket with that answer. I agree with Maarten that either they have a proper plan to stop publishing resources in the BDII and announce it officially and make sure it´s not a problem for their users, or otherwise they should publish proper values.
Thanks.
I don't think I mentioned the ticket number in my earlier email. If it helps, here's the actual ticket:
https://ggus.eu/index.php?mode=ticket_info&ticket_id=113579
Cheers,
Paul.
Hi Maarten, On 08/05/15 02:03, Maarten.Litmaath@cern.ch wrote:
Does anyone know if OSG are planning to drop support for publishing WLCG information via GLUE?
They will not be able to do such a thing very soon AFAIK, as ATLAS, CMS and the WLCG project still have dependencies on the classic info system, even if only to bootstrap various private or derived info systems.
That was certainly my impression, too.
OTOH, OSG have managed to hold off GLUE-2 because none of those stakeholders made a strong case for it so far...
I opened a GGUS ticket describing how some site-level information was wrongly formatted and the ticket was closed as WONT_FIX with the comment that "OSG is deprecating the BDII".
Even if it were true for the long run, it does not mean they can get away with publishing junk already today. However, to get some problem fixed, it may need to be seen as an issue for one of those same stakeholders...
I vaguely remember OSG planning to move towards their own internal information system. I imagine this would have some kind of bridge that would translate this system into GLUE 1.3, acting as a site-level BDII for the entire OSG. However, I could have this wrong. That all said, I absolutely agree that they are using BDII now; even if they are planning to move away from it. In any case, they shouldn't publish broken values and should fix problems when they are pointed out. As usual, just my 2c worth :) Cheers, Paul.
Hi Paul,
I vaguely remember OSG planning to move towards their own internal information system. I imagine this would have some kind of bridge that would translate this system into GLUE 1.3, acting as a site-level BDII for the entire OSG.
That is exactly how they have been publishing their info since many years...
Hi Maarten, On 08/05/15 16:35, Maarten Litmaath wrote:
I vaguely remember OSG planning to move towards their own internal information system. I imagine this would have some kind of bridge that would translate this system into GLUE 1.3, acting as a site-level BDII for the entire OSG.
That is exactly how they have been publishing their info since many years...
Ah, good, so not completely wrong, then :) Here's the other ticket I reported with the same symptom: https://ggus.eu/index.php?mode=ticket_info&ticket_id=113580 The admin did some investigation: the problem does not appear to be due site-local configuration and several OSG sites seem to be affected, so it's possible this is some systemic problem with how OSG publishes into the info-system. Cheers, Paul.
Hi Paul,
Here's the other ticket I reported with the same symptom:
https://ggus.eu/index.php?mode=ticket_info&ticket_id=113580
The admin did some investigation: the problem does not appear to be due site-local configuration and several OSG sites seem to be affected, so it's possible this is some systemic problem with how OSG publishes into the info-system.
Sure, but how much trouble does it cause for whom?
Hi Maarten, On 08/05/15 17:28, Maarten Litmaath wrote:
[...] so it's possible this is some systemic problem with how OSG publishes into the info-system.
Sure, but how much trouble does it cause for whom?
I can't speak for anyone else. I tried to use the info-system; the problem caused some inconvenience as I needed to implement a work-around. However, I think this is really the wrong question. The question should be: does WLCG have any policy on published information? Is it a "Wild West" (where anyone can publish anything they want), or are there some restrictions (e.g., information should be accurate and problems fixed in a timely manner)? Cheers, Paul.
Dear Paul,
Sure, but how much trouble does it cause for whom?
I can't speak for anyone else. I tried to use the info-system; the problem caused some inconvenience as I needed to implement a work-around.
Which inconvenience?
However, I think this is really the wrong question.
The question should be: does WLCG have any policy on published information?
Is it a "Wild West" (where anyone can publish anything they want), or are there some restrictions (e.g., information should be accurate and problems fixed in a timely manner)?
No, sites can´t publish what they want. The glue-validator is validating what the sites publish and sites get a GGUS ticket if what they publish is considered to be an error. This is done for GLUE 2 information. Regards, Maria
Hi Maria, On 11/05/15 11:02, Maria Alandes Pradillo wrote:
I can't speak for anyone else. I tried to use the info-system; the problem caused some inconvenience as I needed to implement a work-around.
Which inconvenience?
Here's the full story. The support period for various dCache versions (2.6.x through to 2.9.x) is about to run out. I've done what I can to make this "well known"; however, there are still ~40% of sites running a version that will run out of support in less than a month. Ultimately, this is the site's choice; however, I wanted to email the site admins directly to make sure they understand the implications of running unsupported software during RUN-2. Using the information-system, I can discover the GlueSiteSysAdminContact for sites running at least one dCache instance with a version that is about to expire. This should provide the information as a URI ("mailto:<address>" format). I found a few sites where the format was wrong: one where the URI was "mailto:mailto:<address>" and a couple where "mailto: <address>" (with a space) was published. Both are pretty trivial to work-around, but I opened GGUS tickets to get the information fixed, as a courtesy to those who might use this information in the future. The site that published "mailto:mailto:<address>" found and fixed the problem quickly. One of the sites that published "mailto: <address>" replied quickly saying WONT_FIX. The other site (publishing "mailto: <address>") investigated, but couldn't see any configuration that would trigger this problem. They also discovered that there are other OSG sites publishing similarly broken URIs, so believes the problem is somehow more general and now doesn't know how to respond to this ticket.
No, sites can´t publish what they want. The glue-validator is validating what the sites publish and sites get a GGUS ticket if what they publish is considered to be an error. This is done for GLUE 2 information.
Of course, I'm in favour of the validator :-) However, there's a technical issue and a deeper problem here. The technical issue is that OSG currently publishes only GLUE 1.3, so the GLUE 2.0 validator will not detect any problems. The deeper issue here is whether sites (or OSG in general) agree to fix problems. If GGUS tickets describing problems are always closed with WONT_FIX then it doesn't matter whether the problem is discovered by an individual or by the validator. HTH, Paul.
Dear Paul,
The support period for various dCache versions (2.6.x through to 2.9.x) is about to run out. I've done what I can to make this "well known"; however, there are still ~40% of sites running a version that will run out of support in less than a month.
We have the MW Officer in WLCG who is responsible for maintaining the MW baseline publishing what MW versions sys admins should be deploying at their sites. We can also advertise this type of information in our Operations meetings. Please, use these channels as they are intended for this purpose. You don´t need to spend time on this. As you may already know, Andrea Manzi is the MW Officer, please, get in touch with him. Anyway, it´s indeed a pity that the contact email of a site is not properly published in the BDII, but I think we need to choose the battles where we want to fight, and especially knowing now that US CMS has plans to get rid of any dependency on the BDII, I´m personally not going to spend any effort to get those fixed. Having said that, we are in touch with OSG so that any possible plan to deprecate the BDII on their side is properly announced and planned, so users know what to expect from the Information System.
Of course, I'm in favour of the validator :-) However, there's a technical issue and a deeper problem here.
The technical issue is that OSG currently publishes only GLUE 1.3, so the GLUE 2.0 validator will not detect any problems.
Exactly, glue-validator is only validating GLUE 2 information, and therefore OSG sites are not validated.
The deeper issue here is whether sites (or OSG in general) agree to fix problems. If GGUS tickets describing problems are always closed with WONT_FIX then it doesn't matter whether the problem is discovered by an individual or by the validator.
If the use case is important enough, I´m sure they will be happy to fix the problem. But as I said before, there are other means to reach sites to make sure they are not running obsolete versions of the MW. Regards, Maria
Hi Maria, Thanks for your reply; some thoughts inline... On 12/05/15 09:36, Maria Alandes Pradillo wrote:
We have the MW Officer in WLCG who is responsible for maintaining the MW baseline publishing what MW versions sys admins should be deploying at their sites. We can also advertise this type of information in our Operations meetings.
Sure. The base-line version was already updated: https://twiki.cern.ch/twiki/bin/view/LCG/WLCGBaselineVersions#table1 but I'll drop Andrea a reminder email.
Anyway, it´s indeed a pity that the contact email of a site is not properly published in the BDII, but I think we need to choose the battles where we want to fight, and especially knowing now that US CMS has plans to get rid of any dependency on the BDII, I´m personally not going to spend any effort to get those fixed.
To be honest, I didn't expect to have to fight OSG to fix published information -- to justify why publishing wrong information is bad and that it should be fixed. On the positive side, one of the tickets has been passed on from the site to OSG software team --- so the problem may still be fixed. Cheers, Paul.
Hi Maria (and Maarten, and others) On 12/05/15 09:36, Maria Alandes Pradillo wrote: [...]
Having said that, we are in touch with OSG so that any possible plan to deprecate the BDII on their side is properly announced and planned, so users know what to expect from the Information System.
I just got the following from Brian via the GGUS ticket #113580:
To be clear - OSG is in the process of deprecating and removing the BDII (that is, having sites no longer show up in the WLCG BDII), in conjunction with our USCMS and USATLAS stakeholders. This has been a multi-year process to remove the BDII requirement from various monitoring, validation, job submission, transfer and accounting systems.
This is starting to show externally-visible differences after multiple years of effort. The USCMS side completed first and we turned-down the first USCMS Tier-2 site last week. More will follow as USCMS coordinates internally. It's still an ongoing effort within USATLAS and we will run the BDII service for as long as a stakeholder needs it.
[...]
So, it looks like USCMS at least are currently in the process of switching off sites, so they will disappear from the info-system. Cheers, Paul.
On Jun 8, 2015, at 1:32 PM, Paul Millar <paul.millar@desy.de> wrote:
So, it looks like USCMS at least are currently in the process of switching off sites, so they will disappear from the info-system.
OSG has been in what appears to e a process of being a self-contained, HTCondor-only shop for some time now. This does not surprise me. It may be the best approach for them, but it does leave open the question of whether any of this has been negotiated or discussed with their WLCG or other partners. It could also be that they are interested in proposing another technology or technologies to replace the features for which they are dropping support. It would be useful to discuss this with them directly. Alan
Dear Paul, Thanks a lot for reporting. I´m also reporting about this at tomorrow´s GDB. If USCMS sites start to disappear from the BDII, we won´t have the whole WLCG picture anymore in the BDII. This mainly affects operational use cases where we use the BDII information to follow up on certain issues. I guess for CMS is fine to stop using the BDII and this is the reason why USCMS sites are dropping it. Let´s see if there are other reactions tomorrow. Regards, Maria
-----Original Message----- From: Paul Millar [mailto:paul.millar@desy.de] Sent: 08 June 2015 20:33 To: Maria Alandes Pradillo; Maarten Litmaath Cc: glue-wg@ogf.org Subject: Re: [glue-wg] OSG dropping GLUE?
Hi Maria (and Maarten, and others)
On 12/05/15 09:36, Maria Alandes Pradillo wrote: [...]
Having said that, we are in touch with OSG so that any possible plan to deprecate the BDII on their side is properly announced and planned, so users know what to expect from the Information System.
I just got the following from Brian via the GGUS ticket #113580:
To be clear - OSG is in the process of deprecating and removing the > BDII (that is, having sites no longer show up in the WLCG BDII), in > conjunction with our USCMS and USATLAS stakeholders. This has been a > multi-year process to remove the BDII requirement from various > monitoring, validation, job submission, transfer and accounting > systems.
This is starting to show externally-visible differences after multiple > years of effort. The USCMS side completed first and we turned-down > the first USCMS Tier-2 site last week. More will follow as USCMS > coordinates internally. It's still an ongoing effort within USATLAS > and we will run the BDII service for as long as a stakeholder needs > it.
[...]
So, it looks like USCMS at least are currently in the process of switching off sites, so they will disappear from the info-system.
Cheers,
Paul.
participants (5)
-
Maarten Litmaath -
Maarten.Litmaath＠cern.ch
-
Maria Alandes Pradillo -
Paul Millar -
Sill, Alan