Hi all, I finally got some time to work on the JSON Rendering. The document now include a formal description of the JSON formats, almost complete (I hope). I have tried to incorporate much of the feedback received so far. Short summary: - Use separate media-types for rendering the different payload contents. Inspired from Florians references to the CDMI spec. - Try to make the single-instance format more lightweight and not duplicating any information available through the discovery interface. Thanks Jamie. - Use the "marker" concept from OpenStack for pagination as suggested by Andy. - "range" and "default" attribute properties as requested by Florian. PDF attached and latex sources committed to gridforge. Comments welcome. Unfortunately I will not be able to attend the plugfest but if you plan a session on discussing the JSON rendering please let me know and I'll call in. regards, Ralf
Hi Ralf, Excellent work you did. Like it I. Comments inline. Am 27.02.2012 um 22:32 schrieb Ralf Nyren:
I have tried to incorporate much of the feedback received so far. Short summary: - Use separate media-types for rendering the different payload contents. Inspired from Florians references to the CDMI spec.
I'm not sure whether I really like the different media types, and I think it could be done without. This would make the content type negotiation easier, if you consider clients that handle different renderings at the same time in conjunction with servers that deliver several different renderings. If we go for HTTP-based usage of the JSON rendering, it should just be application/json. This brings me to the second issue I have: I don't think that it is a good idea to mix JSON data rendering, and the introduction of a JSON-based protocol. I am perfectly fine with the former, but for the latter, we really have HTTP. This might require certain modifications to the HTTP core API (although I wouldn't bet for it without checking), but it would keep things nicely separate. Remember that JSON really is a data format, not a protocol. I am perfectly fine with providing a JSON rendering *for the core model* in terms of data representation, but integrating the protocol would be a similar mistake as it has been done with WS-Agreement, where we are stuck today with WSRF. Besides that, there is no reason forcing to use JSON over HTTP, if they don't want to: replacing the "href" by an URI (and maybe renaming it to "location" would allow for broader use of the JSON data rendering; and everything else is pretty agnostic of HTTP and friends. And btw.: HTTP gives us all we need to do the protocol part; the RESTfulness lies in there, anyway. Therefore, the interaction part should be described there, agnostic of the content-type (after all, that's what HTTP is about). That way, people could implement the HTTP protocol rendering once, and reuse it for several OCCI data renderings. Let's not do the same mistake we did for the HTTP rendering yet again...
- Try to make the single-instance format more lightweight and not duplicating any information available through the discovery interface. Thanks Jamie.
That actually I like very much. It might be nice to think about a shallow/deep rendering option on the protocol level, but that's just a thought.
- Use the "marker" concept from OpenStack for pagination as suggested by Andy.
This is also something that should be taken care of in the protocol, rather than the data format. Why on earth would people want to see this if the protocol they use only delivers batches of things? And in fact, looking at the spec, it has been done on the HTTP level, anyway. Frankly, this should be something to be added to the HTTP protocol rendering and made available to all data representations, rather than in a proprietary, JSON-specific way only.
- "range" and "default" attribute properties as requested by Florian.
Great idea. We might want to consider pushing this upstream to core. Ready to be clobbered by the mongol hordes… -A. -- Alexander Papaspyrou alexander.papaspyrou@tu-dortmund.de
On Wed, 29 Feb 2012 09:36:50 +0100, <alexander.papaspyrou@tu-dortmund.de> wrote:
I have tried to incorporate much of the feedback received so far. Short summary: - Use separate media-types for rendering the different payload contents. Inspired from Florians references to the CDMI spec.
I'm not sure whether I really like the different media types, and I think it could be done without. This would make the content type negotiation easier, if you consider clients that handle different renderings at the same time in conjunction with servers that deliver several different renderings.
Sure, the media-type negotiation will involve a few more bytes in the Accept header but I don't see that as such a big deal. But lets leave that issue for the moment and just look at the payload _received_ by a server or client. The HTTP protocol's use of Accept and Content-Type headers basically work like this: Client: Give me the content of this URL, I can handle (Accept) this list of different formats (media-types) and I really prefer you to use format X or Y if you can, please. Server: Look at content pointed to by URL and try to return something that satisfies the client. Client: Look at the Content-Type and apply decoding/processing based on media-type alone. So, essentially the Client do not know what the Server will return. The media-type should be enough to tell the Client what to do with the payload. I rather have it that the Client does not have to poke around within the payload to figure out what to do with it (e.g. the html rendering mess).
If we go for HTTP-based usage of the JSON rendering, it should just be application/json.
No, I disagree. By saying application/json the receiver only knows about the syntax. Let's say you get a bunch of data with media-type application/json. Ok, you can decode the json data but then what? You have no clue about the semantics of the format. I believe there is a reason that the Atom Syndication Format use media type application/atom+xml and not just application/xml. If you happen to be a browser you just throw the json structures at the user and let him figure out what it means. For OCCI however your typical client may be a machine and then it needs to know how to interpret the json structures. So at a minimum we should say application/occi+json. But in that case we need something in the json structures to say e.g. "format": "entity" vs "format": "discover" in order to distinguish between the different data representation formats. And no, I don't want the one-format-fits-all we did for the HTTP rendering. Therefore I vote on having separate media-types for each data representation format.
This brings me to the second issue I have: I don't think that it is a good idea to mix JSON data rendering, and the introduction of a JSON-based protocol. I am perfectly fine with the former, but for the latter, we really have HTTP. This might require certain modifications to the HTTP core API (although I wouldn't bet for it without checking), but it would keep things nicely separate.
This is a very good point, I agree we should avoid creating a JSON protocol. However, we do not have a clean OCCI HTTP Protocol spec today. The HTTP Rendering doc is a happy mix of both protocol and data format stuff. Since much of the multi-resource support was added in a hurry just before public comment there are things in there that does not fully work with the JSON rendering. No big issues a client would be affected by but enough that I cannot use it as a basis for the JSON rendering spec. If people do not disagree I would be glad to create a clean "OCCI HTTP Protocol" doc totally free of any payload or header rendering data formats. Then the JSON and text/xxx renderings could be built on top of it. I.e. a clean separation between data format and protocol. We could call it occi/1.2 and have it backward compatible with occi/1.1. Any objections to that? ;)
Besides that, there is no reason forcing to use JSON over HTTP, if they don't want to: replacing the "href" by an URI (and maybe renaming it to "location" would allow for broader use of the JSON data rendering; and everything else is pretty agnostic of HTTP and friends.
Ok, renaming "href" back to "uri" then. Good proof you actually read the details :D (mind-note to add other small obscure changes in next rev just to see if people notice...) What to call the category identifier? I used "rel" now based on the HTTP way of doing things. Any better naming suggestions, "type" ? [1]
And btw.: HTTP gives us all we need to do the protocol part; the RESTfulness lies in there, anyway. Therefore, the interaction part should be described there, agnostic of the content-type (after all, that's what HTTP is about). That way, people could implement the HTTP protocol rendering once, and reuse it for several OCCI data renderings.
Agreed on that. Support my proposal on an OCCI HTTP Protocol doc above :)
- Try to make the single-instance format more lightweight and not duplicating any information available through the discovery interface. Thanks Jamie.
That actually I like very much. It might be nice to think about a shallow/deep rendering option on the protocol level, but that's just a thought.
If you are referring to all the white-space in the examples it is just there for readability. I should put in a note that a server/client should remove all unnecessary white-space for real communication. If not, please elaborate.
- Use the "marker" concept from OpenStack for pagination as suggested by Andy.
This is also something that should be taken care of in the protocol, rather than the data format. Why on earth would people want to see this if the protocol they use only delivers batches of things? And in fact, looking at the spec, it has been done on the HTTP level, anyway.
It is done both on the HTTP level (query parameters) and in the payload :-\ Is it just the "next" and "limit" members you want removed from the collection format? Anything else you deem to be protocol stuff? I guess we could put the "next" link in a Link header and actually use RFC5988 as it was meant to be used (as opposed to how we abused it in text/occi rendering). We have to verify if browsers would choke on this though and it makes life a bit harder for JavaScript implementations which would need to parse the Link headers. Other ideas?
Frankly, this should be something to be added to the HTTP protocol rendering and made available to all data representations, rather than in a proprietary, JSON-specific way only.
Agreed, with reservation for possible browser problems...
- "range" and "default" attribute properties as requested by Florian.
Great idea. We might want to consider pushing this upstream to core.
That was my intention. I have long wanted to have the "default" property into Core since without it a client cannot create custom templates. Thanks for the feedback Alex. More of this and we might have something to publish in not a too distant future. /Ralf [1] http://martinfowler.com/bliki/TwoHardThings.html
On 2/29/2012 4:11 AM, Ralf Nyren wrote:
On Wed, 29 Feb 2012 09:36:50 +0100, <alexander.papaspyrou@tu-dortmund.de> wrote:
I have tried to incorporate much of the feedback received so far. Short summary: - Use separate media-types for rendering the different payload contents. Inspired from Florians references to the CDMI spec.
I'm not sure whether I really like the different media types, and I think it could be done without. This would make the content type negotiation easier, if you consider clients that handle different renderings at the same time in conjunction with servers that deliver several different renderings.
Sure, the media-type negotiation will involve a few more bytes in the Accept header but I don't see that as such a big deal. But lets leave that issue for the moment and just look at the payload _received_ by a server or client.
The HTTP protocol's use of Accept and Content-Type headers basically work like this: Client: Give me the content of this URL, I can handle (Accept) this list of different formats (media-types) and I really prefer you to use format X or Y if you can, please. Server: Look at content pointed to by URL and try to return something that satisfies the client. Client: Look at the Content-Type and apply decoding/processing based on media-type alone.
So, essentially the Client do not know what the Server will return. The media-type should be enough to tell the Client what to do with the payload. I rather have it that the Client does not have to poke around within the payload to figure out what to do with it (e.g. the html rendering mess).
The fact that the client doesn't know what the server will return is incidental. If the client couldn't discriminate and handle the formats, it shouldn't be requesting them in the accept header. The challenge here is whether there is sufficient information returned in the payload for the client to distinguish the content, which I'm not too sure is relevant. Since JSON is a data format and the returned content is associated with an occi applications's taxonomy and ontology, their needs to way, explicitly defined or implicate to client/server relationship, for the client to understand the returned information. It can be argued that the nature of http over tcp is sequential and lock stepped by definition. So, the client should be 'expecting' a known taxonomy based on the request(s) issued. I didn't notice any capabilities supporting for multiple queued requests on a connection and any capabilities supporting out of order request completion on the same connection. Based on the traditional nature of the http client/server protocol, saying the media type is json should be sufficient. If at some time in the future, request queuing on a connection is supported, depending on method implemented it should be revisited.
If we go for HTTP-based usage of the JSON rendering, it should just be application/json.
No, I disagree. By saying application/json the receiver only knows about the syntax. Let's say you get a bunch of data with media-type application/json. Ok, you can decode the json data but then what? You have no clue about the semantics of the format. I believe there is a reason that the Atom Syndication Format use media type application/atom+xml and not just application/xml.
If you happen to be a browser you just throw the json structures at the user and let him figure out what it means. For OCCI however your typical client may be a machine and then it needs to know how to interpret the json structures.
So at a minimum we should say application/occi+json. But in that case we need something in the json structures to say e.g. "format": "entity" vs "format": "discover" in order to distinguish between the different data representation formats. And no, I don't want the one-format-fits-all we did for the HTTP rendering. Therefore I vote on having separate media-types for each data representation format.
This brings me to the second issue I have: I don't think that it is a good idea to mix JSON data rendering, and the introduction of a JSON-based protocol. I am perfectly fine with the former, but for the latter, we really have HTTP. This might require certain modifications to the HTTP core API (although I wouldn't bet for it without checking), but it would keep things nicely separate.
This is a very good point, I agree we should avoid creating a JSON protocol.
However, we do not have a clean OCCI HTTP Protocol spec today. The HTTP Rendering doc is a happy mix of both protocol and data format stuff. Since much of the multi-resource support was added in a hurry just before public comment there are things in there that does not fully work with the JSON rendering. No big issues a client would be affected by but enough that I cannot use it as a basis for the JSON rendering spec.
If people do not disagree I would be glad to create a clean "OCCI HTTP Protocol" doc totally free of any payload or header rendering data formats. Then the JSON and text/xxx renderings could be built on top of it. I.e. a clean separation between data format and protocol. We could call it occi/1.2 and have it backward compatible with occi/1.1. Any objections to that? ;)
Besides that, there is no reason forcing to use JSON over HTTP, if they don't want to: replacing the "href" by an URI (and maybe renaming it to "location" would allow for broader use of the JSON data rendering; and everything else is pretty agnostic of HTTP and friends.
Ok, renaming "href" back to "uri" then. Good proof you actually read the details :D (mind-note to add other small obscure changes in next rev just to see if people notice...)
What to call the category identifier? I used "rel" now based on the HTTP way of doing things. Any better naming suggestions, "type" ? [1]
A little perspective on 'rel'. This is an artifact based on the HTML5 and RDFa rendering that haven't yet matured. I would suggest we keep the 'rel' in case the there is a huge shift to semantic approaches for cloud computing.
And btw.: HTTP gives us all we need to do the protocol part; the RESTfulness lies in there, anyway. Therefore, the interaction part should be described there, agnostic of the content-type (after all, that's what HTTP is about). That way, people could implement the HTTP protocol rendering once, and reuse it for several OCCI data renderings.
Agreed on that. Support my proposal on an OCCI HTTP Protocol doc above :)
- Try to make the single-instance format more lightweight and not duplicating any information available through the discovery interface. Thanks Jamie.
That actually I like very much. It might be nice to think about a shallow/deep rendering option on the protocol level, but that's just a thought.
If you are referring to all the white-space in the examples it is just there for readability. I should put in a note that a server/client should remove all unnecessary white-space for real communication.
If not, please elaborate.
I'm interested in this as well.
- Use the "marker" concept from OpenStack for pagination as suggested by Andy.
This is also something that should be taken care of in the protocol, rather than the data format. Why on earth would people want to see this if the protocol they use only delivers batches of things? And in fact, looking at the spec, it has been done on the HTTP level, anyway.
It is done both on the HTTP level (query parameters) and in the payload :-\
Is it just the "next" and "limit" members you want removed from the collection format? Anything else you deem to be protocol stuff?
I guess we could put the "next" link in a Link header and actually use RFC5988 as it was meant to be used (as opposed to how we abused it in text/occi rendering). We have to verify if browsers would choke on this though and it makes life a bit harder for JavaScript implementations which would need to parse the Link headers. Other ideas?
This is a query capability and should not be part of any portion of the format or http protocol. I think there is some confusion about document structure and presentation with the occi model. Granted, this has been clearly vetted withing the group. We should keep this discussion in terms of a query. The "marker" as suggested is essentially acting as an arbitrary index or iteration context in application. We should be clear on the use cases and whether the "marker/iterator' applies to occi entities, json entities and payload size. There are very separate concerns that seem to be getting integrated as some levels.
Frankly, this should be something to be added to the HTTP protocol rendering and made available to all data representations, rather than in a proprietary, JSON-specific way only.
Agreed, with reservation for possible browser problems...
- "range" and "default" attribute properties as requested by Florian.
Great idea. We might want to consider pushing this upstream to core.
That was my intention. I have long wanted to have the "default" property into Core since without it a client cannot create custom templates.
Thanks for the feedback Alex. More of this and we might have something to publish in not a too distant future.
/Ralf
[1] http://martinfowler.com/bliki/TwoHardThings.html _______________________________________________ occi-wg mailing list occi-wg@ogf.org https://www.ogf.org/mailman/listinfo/occi-wg
On Wed, 29 Feb 2012 18:32:18 +0100, Gary Mazz <garymazzaferro@gmail.com> wrote:
Based on the traditional nature of the http client/server protocol, saying the media type is json should be sufficient. If at some time in the future, request queuing on a connection is supported, depending on method implemented it should be revisited.
Not convinced. Let's say I want to write a nice litte firefox plugin which can render OCCI JSON formats nicely. If the media-type does not tell that the content is OCCI-json and not some arbitrary json, such a plugin would be hard to get working. The JSON rendering is exposed over HTTP. Any HTTP client is able to issue valid requests. If the payload returned is marked as "occi-stuff" it is much easier to have support in a wider range of clients. Whether to have just application/occi+json or multiple media-types for json is a different topic though.
What to call the category identifier? I used "rel" now based on the HTTP way of doing things. Any better naming suggestions, "type" ? [1]
A little perspective on 'rel'. This is an artifact based on the HTML5 and RDFa rendering that haven't yet matured. I would suggest we keep the 'rel' in case the there is a huge shift to semantic approaches for cloud computing.
Sure, keep "rel" then. Other thoughts, someone?
I guess we could put the "next" link in a Link header and actually use RFC5988 as it was meant to be used (as opposed to how we abused it in text/occi rendering). We have to verify if browsers would choke on this though and it makes life a bit harder for JavaScript implementations which would need to parse the Link headers. Other ideas?
This is a query capability and should not be part of any portion of the format or http protocol.
Where to put it then? The whole point of the "marker" concept is to return a link to the next "page" in the response. That link has to go somewhere...
I think there is some confusion about document structure and presentation with the occi model. Granted, this has been clearly vetted withing the group. We should keep this discussion in terms of a query. The "marker" as suggested is essentially acting as an arbitrary index or iteration context in application. We should be clear on the use cases and whether the "marker/iterator' applies to occi entities, json entities and payload size. There are very separate concerns that seem to be getting integrated as some levels.
The marker applies to resource instances (Entity sub-type instances ;) but on a HTTP protocol level. Makes sense? /Ralf
On 2/29/2012 11:07 AM, Ralf Nyren wrote:
On Wed, 29 Feb 2012 18:32:18 +0100, Gary Mazz <garymazzaferro@gmail.com> wrote:
Based on the traditional nature of the http client/server protocol, saying the media type is json should be sufficient. If at some time in the future, request queuing on a connection is supported, depending on method implemented it should be revisited.
Not convinced. Let's say I want to write a nice litte firefox plugin which can render OCCI JSON formats nicely. If the media-type does not tell that the content is OCCI-json and not some arbitrary json, such a plugin would be hard to get working.
The JSON rendering is exposed over HTTP. Any HTTP client is able to issue valid requests. If the payload returned is marked as "occi-stuff" it is much easier to have support in a wider range of clients.
Whether to have just application/occi+json or multiple media-types for json is a different topic though.
Well, I think the root of this discussion is whether the client knows he is talking to occi or not. We already know the json specification does not include the capability to presenting occi information in a webpage. This is important for the context of the discussion. So, what other types of 'generic' applications are going to be talking to occi ? Web browser, probably not. SEO robots ? maybe.. Other applications that know about occi, definitely. We need to consider applying the client's context in defining the capabilities. OCCI is an application specific interface with a narrow functional scope. It does not 'need', based on today's client/server operation, capabilities as if it were serving generic web pages. If the specification included html5 and widgets, I would be prone to agree with your position. However, unless there is a need in the future to expand the capabilities, the current approach on this subject seems unnecessary and adds additional complication to client implementations.
What to call the category identifier? I used "rel" now based on the HTTP way of doing things. Any better naming suggestions, "type" ? [1]
A little perspective on 'rel'. This is an artifact based on the HTML5 and RDFa rendering that haven't yet matured. I would suggest we keep the 'rel' in case the there is a huge shift to semantic approaches for cloud computing.
Sure, keep "rel" then. Other thoughts, someone?
I guess we could put the "next" link in a Link header and actually use RFC5988 as it was meant to be used (as opposed to how we abused it in text/occi rendering). We have to verify if browsers would choke on this though and it makes life a bit harder for JavaScript implementations which would need to parse the Link headers. Other ideas?
This is a query capability and should not be part of any portion of the format or http protocol.
Where to put it then? The whole point of the "marker" concept is to return a link to the next "page" in the response. That link has to go somewhere...
This is the confusion. The link to the next page is a "query" and not part of the occi data. It is important but for other reasons. In the http rendering, it was required to place it where it was, there was not other capability available. Now that there was time to think about the approach in more detail and take advantage of json's additional capabilities, we may want to reconsider the approach. We have 2 issues with the current query mechanism, query context and data consistency. We originally perused "links to the next page" as part of the HTML5 rendering. I was in support of the idea at the time because it made web client development easier. Since that time, the industry has changed and browsers are smarter and better techniques including MVC have entered the space. The occi http specification retained some of the html5 artifacts. An argument against links to the next page is that the link/iterator is a state resulting from a query maintained in the server. Since the client has already requested a range, the client should maintain the position as part of the query context passed in the query request. Otherwise, the protocol between the client an server is no longer stateless. Additionally, if the client is already maintaining the query context, placing a query context in the server is only redundant.
I think there is some confusion about document structure and presentation with the occi model. Granted, this has been clearly vetted withing the group. We should keep this discussion in terms of a query. The "marker" as suggested is essentially acting as an arbitrary index or iteration context in application. We should be clear on the use cases and whether the "marker/iterator' applies to occi entities, json entities and payload size. There are very separate concerns that seem to be getting integrated as some levels.
The marker applies to resource instances (Entity sub-type instances ;) but on a HTTP protocol level. Makes sense?
New comment: Multi-user administration consoles and automated has introduced new complexities. Configurations and states can change after a page has been transferred to the client. Data consistency of both the consumer and provider occi data may change after the consumer has queried. It could be unreasonable for the provider to maintain to event information in occi's context. If the notification is tied to a query and the life cycle of the query is tied to the connection, the provider can discard the query and notification when the connection is closed. There need to be either a notification or a time tag assigned to a query to detect changes on state.
/Ralf _______________________________________________ occi-wg mailing list occi-wg@ogf.org https://www.ogf.org/mailman/listinfo/occi-wg
On Wed, 29 Feb 2012 14:36:16 -0700, Gary Mazz <garymazzaferro@gmail.com> wrote:
Well, I think the root of this discussion is whether the client knows he
is talking to occi or not. We already know the json specification does not include the capability to presenting occi information in a webpage. This is important for the context of the discussion. So, what other types of 'generic' applications are going to be talking to occi ? Web browser, probably not. SEO robots ? maybe.. Other applications that
know about occi, definitely. We need to consider applying the client's context in defining the capabilities. OCCI is an application specific interface with a narrow functional scope. It does not 'need', based on today's client/server operation, capabilities as if it were serving generic web pages. If the specification included html5 and widgets, I would be prone to agree with your position. However, unless there is a need in the future to expand the capabilities, the current approach on this subject seems unnecessary and adds additional complication to client implementations.
Just to make clear, this argument is whether to use application/json or application/occi+json. Not about using multiple media-types as in my latest proposal. Yes, agreed that most occi clients will be aware that they talk to an OCCI server and not some other json capable service. However, I don't see what the complications would be of having application/occi+json. You say this may be relevant in the future so why not get it right from the beginning, the penalty seems very low to me. A valid use case for OCCI Json rendering on the browser side is an AJAX client as a generic management interface for arbitrary OCCI services. However, you could of course argue this is an application aware of talking to an OCCI server. Additionally you actually avoid some potential security problems by not using "applcation/json". E.g. silly javascript eval() whenever json content is found.
Where to put it then? The whole point of the "marker" concept is to return a link to the next "page" in the response. That link has to go somewhere...
An argument against links to the next page is that the link/iterator is a state resulting from a query maintained in the server. Since the client has already requested a range, the client should maintain the position as part of the query context passed in the query request. Otherwise, the protocol between the client an server is no longer stateless. Additionally, if the client is already maintaining the query context, placing a query context in the server is only redundant.
Good point, thanks for taking time to explain. We simply leave it up to the client to maintain state on which "page" it has requested last. Protocol-wise paging only exist on the query level then. I would like to leave the "total collection size" value in the data format though. That way a server still has the option to throttle the response size when necessary. Something like this then? "Get 50-entries page _after_ resource xxx": GET /compute/?marker=xxx&start=1&end=51 "Get 50-entries page _before_ resource xxx": GET /compute/?marker=xxx&start=-1&end=-51 "Get 10 resources _including_ resource xxx": GET / compute/?marker=xxx&start=0&end=10
New comment:
Multi-user administration consoles and automated has introduced new complexities. Configurations and states can change after a page has been
transferred to the client. Data consistency of both the consumer and provider occi data may change after the consumer has queried.
It could be unreasonable for the provider to maintain to event information in occi's context. If the notification is tied to a query and the life cycle of the query is tied to the connection, the provider can discard the query and notification when the connection is closed. There need to be either a notification or a time tag assigned to a query
to detect changes on state.
Yes, the currently proposed pagination would produce inconsistent results if the collection is modified between requests. The thought was to accept this as a limitation and say "a provider SHOULD by default sort a collection based on resource creation time". Adding support for server side "collection pagination state" as you suggest would be nice from a client perspective but the question is if the added complexity on the server side is really worth it. regards, Ralf
Am 01.03.2012 um 12:44 schrieb Ralf Nyren:
On Wed, 29 Feb 2012 14:36:16 -0700, Gary Mazz <garymazzaferro@gmail.com> wrote:
Good point, thanks for taking time to explain. We simply leave it up to the client to maintain state on which "page" it has requested last. Protocol-wise paging only exist on the query level then.
I would like to leave the "total collection size" value in the data format though. That way a server still has the option to throttle the response size when necessary.
Something like this then?
"Get 50-entries page _after_ resource xxx": GET /compute/?marker=xxx&start=1&end=51
"Get 50-entries page _before_ resource xxx": GET /compute/?marker=xxx&start=-1&end=-51
"Get 10 resources _including_ resource xxx": GET / compute/?marker=xxx&start=0&end=10
More or less. We have to check inhowfar this needs to be done as parts of the query paramaters, and how this can be sensibly combined with RFC5988 information. -A. -- Alexander Papaspyrou alexander.papaspyrou@tu-dortmund.de
On 3/1/2012 4:44 AM, Ralf Nyren wrote:
On Wed, 29 Feb 2012 14:36:16 -0700, Gary Mazz<garymazzaferro@gmail.com> wrote:
Well, I think the root of this discussion is whether the client knows he is talking to occi or not. We already know the json specification does not include the capability to presenting occi information in a webpage. This is important for the context of the discussion. So, what other types of 'generic' applications are going to be talking to occi ? Web browser, probably not. SEO robots ? maybe.. Other applications that know about occi, definitely. We need to consider applying the client's context in defining the capabilities. OCCI is an application specific interface with a narrow functional scope. It does not 'need', based on today's client/server operation, capabilities as if it were serving generic web pages. If the specification included html5 and widgets, I would be prone to agree with your position. However, unless there is a need in the future to expand the capabilities, the current approach on this subject seems unnecessary and adds additional complication to client implementations. Just to make clear, this argument is whether to use application/json or application/occi+json. Not about using multiple media-types as in my latest proposal.
Yes, agreed that most occi clients will be aware that they talk to an OCCI server and not some other json capable service.
However, I don't see what the complications would be of having application/occi+json. You say this may be relevant in the future so why not get it right from the beginning, the penalty seems very low to me.
A valid use case for OCCI Json rendering on the browser side is an AJAX client as a generic management interface for arbitrary OCCI services. However, you could of course argue this is an application aware of talking to an OCCI server. Additionally you actually avoid some potential security problems by not using "applcation/json". E.g. silly javascript eval() whenever json content is found.
I was just working through the use cases to evaluate the impact of 'application/json' and 'application/occi+json'. It seems, at least from a simple rationalization, that the client would know it talking to an occi server. In that case, it probably doesn't matter which media type is presented. The second use case, is someone using a browser to access the occi/json site. In that case, its probably a developer or technically skilled person and they would also know they are accessing an occi server. Again, it probably doesn't matter which media type is presented. The last use case is a robot, ie seo scanner. They do request media types as 'application/json'. The question is do we want to refuse those requests via media types ?
Where to put it then? The whole point of the "marker" concept is to return a link to the next "page" in the response. That link has to go somewhere... An argument against links to the next page is that the link/iterator is a state resulting from a query maintained in the server. Since the client has already requested a range, the client should maintain the position as part of the query context passed in the query request. Otherwise, the protocol between the client an server is no longer stateless. Additionally, if the client is already maintaining the query context, placing a query context in the server is only redundant. Good point, thanks for taking time to explain. We simply leave it up to the client to maintain state on which "page" it has requested last. Protocol-wise paging only exist on the query level then.
I would like to leave the "total collection size" value in the data format though. That way a server still has the option to throttle the response size when necessary.
Something like this then?
"Get 50-entries page _after_ resource xxx": GET /compute/?marker=xxx&start=1&end=51
"Get 50-entries page _before_ resource xxx": GET /compute/?marker=xxx&start=-1&end=-51
"Get 10 resources _including_ resource xxx": GET / compute/?marker=xxx&start=0&end=10
Yes, this will work nicely :)
New comment:
Multi-user administration consoles and automated has introduced new complexities. Configurations and states can change after a page has been transferred to the client. Data consistency of both the consumer and provider occi data may change after the consumer has queried.
It could be unreasonable for the provider to maintain to event information in occi's context. If the notification is tied to a query and the life cycle of the query is tied to the connection, the provider can discard the query and notification when the connection is closed. There need to be either a notification or a time tag assigned to a query to detect changes on state. Yes, the currently proposed pagination would produce inconsistent results if the collection is modified between requests. The thought was to accept this as a limitation and say "a provider SHOULD by default sort a
collection based on resource creation time".
The issue is 'creation time', which seems like a good idea. But, there is more than creation to be concerned with..
Adding support for server side "collection pagination state" as you suggest would be nice from a client perspective but the question is if the added complexity on the server side is really worth it.
From a purely interface perspective, I would also challenge the value of the approach. However, from a client usability perspective, I can't see not having some push event indicating changes. The alternative forces the client to continuously poll the server. Multi-access provider interfaces require a way for clients to keep occi information consistent across the multiple occi clients. This use case becomes increasing important for brokers. This type of query is an interesting case for occi's approach to interface. A notification query is a long running occi operation. Currently, states of long running operations associated with occi objects are maintained with the occi object. This feature allows the interface to remain stateless which greatly simplifies client and server implementations. In the past, long running occi operations has been consciously avoided, outside of some recommendations. We probably are running into a time where 'long running operations' and push indicators are revisited. cheers, gary
regards, Ralf
Am 29.02.2012 um 19:07 schrieb Ralf Nyren:
On Wed, 29 Feb 2012 18:32:18 +0100, Gary Mazz <garymazzaferro@gmail.com> wrote:
Based on the traditional nature of the http client/server protocol, saying the media type is json should be sufficient. If at some time in the future, request queuing on a connection is supported, depending on method implemented it should be revisited.
Not convinced. Let's say I want to write a nice litte firefox plugin which can render OCCI JSON formats nicely. If the media-type does not tell that the content is OCCI-json and not some arbitrary json, such a plugin would be hard to get working.
The JSON rendering is exposed over HTTP. Any HTTP client is able to issue valid requests. If the payload returned is marked as "occi-stuff" it is much easier to have support in a wider range of clients.
Whether to have just application/occi+json or multiple media-types for json is a different topic though.
I see your point. I guess then we should have application/occi+<something>, and then really stay with that.
I guess we could put the "next" link in a Link header and actually use RFC5988 as it was meant to be used (as opposed to how we abused it in text/occi rendering). We have to verify if browsers would choke on this though and it makes life a bit harder for JavaScript implementations which would need to parse the Link headers. Other ideas?
This is a query capability and should not be part of any portion of the format or http protocol.
Where to put it then? The whole point of the "marker" concept is to return a link to the next "page" in the response. That link has to go somewhere…
Well, I think queries should be handled via URL parameters. But I thought we were talking about pagination interlinking.
I think there is some confusion about document structure and presentation with the occi model. Granted, this has been clearly vetted withing the group. We should keep this discussion in terms of a query. The "marker" as suggested is essentially acting as an arbitrary index or iteration context in application. We should be clear on the use cases and whether the "marker/iterator' applies to occi entities, json entities and payload size. There are very separate concerns that seem to be getting integrated as some levels.
The marker applies to resource instances (Entity sub-type instances ;) but on a HTTP protocol level. Makes sense?
If you would use it in the context of RFC5988, then yes. -A.
/Ralf _______________________________________________ occi-wg mailing list occi-wg@ogf.org https://www.ogf.org/mailman/listinfo/occi-wg
-- Alexander Papaspyrou alexander.papaspyrou@tu-dortmund.de
Hi Gary, Ralf, Am 29.02.2012 um 18:32 schrieb Gary Mazz:
On 2/29/2012 4:11 AM, Ralf Nyren wrote:
On Wed, 29 Feb 2012 09:36:50 +0100, <alexander.papaspyrou@tu-dortmund.de> wrote:
Besides that, there is no reason forcing to use JSON over HTTP, if they don't want to: replacing the "href" by an URI (and maybe renaming it to "location" would allow for broader use of the JSON data rendering; and everything else is pretty agnostic of HTTP and friends.
Ok, renaming "href" back to "uri" then. Good proof you actually read the details :D (mind-note to add other small obscure changes in next rev just to see if people notice...)
What to call the category identifier? I used "rel" now based on the HTTP way of doing things. Any better naming suggestions, "type" ? [1]
A little perspective on 'rel'. This is an artifact based on the HTML5 and RDFa rendering that haven't yet matured. I would suggest we keep the 'rel' in case the there is a huge shift to semantic approaches for cloud computing.
Yup, agreed. That's also what RFC5988 uses.
And btw.: HTTP gives us all we need to do the protocol part; the RESTfulness lies in there, anyway. Therefore, the interaction part should be described there, agnostic of the content-type (after all, that's what HTTP is about). That way, people could implement the HTTP protocol rendering once, and reuse it for several OCCI data renderings.
Agreed on that. Support my proposal on an OCCI HTTP Protocol doc above :)
- Try to make the single-instance format more lightweight and not duplicating any information available through the discovery interface. Thanks Jamie.
That actually I like very much. It might be nice to think about a shallow/deep rendering option on the protocol level, but that's just a thought.
If you are referring to all the white-space in the examples it is just there for readability. I should put in a note that a server/client should remove all unnecessary white-space for real communication.
If not, please elaborate.
I'm interested in this as well.
Nope. I meant whether, for JSON, one might want to get a resource with its dependencies, rendered inline. Thinking about it overnight however led to the conclusion that I'd better be ignored when proposing such things...
- Use the "marker" concept from OpenStack for pagination as suggested by Andy.
This is also something that should be taken care of in the protocol, rather than the data format. Why on earth would people want to see this if the protocol they use only delivers batches of things? And in fact, looking at the spec, it has been done on the HTTP level, anyway.
It is done both on the HTTP level (query parameters) and in the payload :-\
Is it just the "next" and "limit" members you want removed from the collection format? Anything else you deem to be protocol stuff?
I guess we could put the "next" link in a Link header and actually use RFC5988 as it was meant to be used (as opposed to how we abused it in text/occi rendering). We have to verify if browsers would choke on this though and it makes life a bit harder for JavaScript implementations which would need to parse the Link headers. Other ideas?
This is a query capability and should not be part of any portion of the format or http protocol.
I think there is some confusion about document structure and presentation with the occi model. Granted, this has been clearly vetted withing the group. We should keep this discussion in terms of a query. The "marker" as suggested is essentially acting as an arbitrary index or iteration context in application. We should be clear on the use cases and whether the "marker/iterator' applies to occi entities, json entities and payload size. There are very separate concerns that seem to be getting integrated as some levels.
I think the link header is a nice approach for this. They also allow for indexes, anyway. @Gary: I thought of using them as a means of orientation for pagination. But that would also be something to be done on the HTTP spec level, and not specifically for JSON. And since we are at it, we should also consider moving queries to the query fragment of URLs.
Frankly, this should be something to be added to the HTTP protocol rendering and made available to all data representations, rather than in a proprietary, JSON-specific way only.
Agreed, with reservation for possible browser problems…
Well, the base HTTP API is beyond what browsers can handle, anyway.
- "range" and "default" attribute properties as requested by Florian.
Great idea. We might want to consider pushing this upstream to core.
That was my intention. I have long wanted to have the "default" property into Core since without it a client cannot create custom templates.
Yup. Very good point.
Thanks for the feedback Alex. More of this and we might have something to publish in not a too distant future.
Hope so ;-) -A. -- Alexander Papaspyrou alexander.papaspyrou@tu-dortmund.de
Hi Ralf, at todays Plugfest Jamie Marshall, David Slik, Houssem Medhioub, Andy Edmonds, Thijs Metsch and I discussed about the JSON rendering. Your proposal was well received. For the entity rendering there was the suggestion to make it more intuitively usable. The main difference is, that the attributes should be directly accessible by name structure. e.g. make it possible to do the following commands in your favorite programming language: var occi = GET /compute/123 occi.kind occi.compute.cores occi.my_mixin.my_attribute The only thing we have to prohibit then is, that a single service provider has mixins from different domains with the same term and attributes. In the category rendering we proposed to use special parameters for different attribute types. e.g. having min_length and max_length for strings and an array of bounds for integers or floats. Here are some ideas: http://pastebin.com/y2K7bu1S http://pastebin.com/KLF4YKny What do you think about them? Am 27.02.2012 um 22:32 schrieb Ralf Nyren:
Hi all,
I finally got some time to work on the JSON Rendering. The document now include a formal description of the JSON formats, almost complete (I hope).
I have tried to incorporate much of the feedback received so far. Short summary: - Use separate media-types for rendering the different payload contents. Inspired from Florians references to the CDMI spec. - Try to make the single-instance format more lightweight and not duplicating any information available through the discovery interface. Thanks Jamie. - Use the "marker" concept from OpenStack for pagination as suggested by Andy. - "range" and "default" attribute properties as requested by Florian.
PDF attached and latex sources committed to gridforge.
Comments welcome.
Unfortunately I will not be able to attend the plugfest but if you plan a session on discussing the JSON rendering please let me know and I'll call in.
regards, Ralf<json_rendering.pdf>_______________________________________________ occi-wg mailing list occi-wg@ogf.org https://www.ogf.org/mailman/listinfo/occi-wg
On Wed, 29 Feb 2012 12:23:25 +0100, Feldhaus, Florian <florian.feldhaus@gwdg.de> wrote:
Hi Ralf,
Hi Florian and thanks for the update!
at todays Plugfest Jamie Marshall, David Slik, Houssem Medhioub, Andy Edmonds, Thijs Metsch and I discussed about the JSON rendering. Your proposal was well received. For the entity rendering there was the suggestion to make it more intuitively usable. The main difference is, that the attributes should be directly accessible by name structure. e.g. make it possible to do the following commands in your favorite programming language: var occi = GET /compute/123 occi.kind occi.compute.cores occi.my_mixin.my_attribute
Yes, would be really nice if we could get something like that to work.
The only thing we have to prohibit then is, that a single service provider has mixins from different domains with the same term and attributes.
This is the hard part. I cannot see a way to do this without changing fundamental parts of OCCI Core :-\ Today we have 2 name-spaces where conflicts could arise: 1. Entity sub-types 2. Attributes 1: For entity sub-types we have a categorisation schema (as part of the Kind/Mixin identifier) which allows arbitrary naming within the schema. E.g. you could have 100 providers interacting and sharing resources which each of them have their own "compute" type (term="compute"). It works since they are forced to have unique categorisation schemas. Let's assume for a minute that we would drop the categorisation schema and only have the "term" left. We prohibit anyone from using resource, link, compute, storage, etc since we have defined them already. Later on we want to add a new infrastructure type but since it was not forbidden from the beginning adding a new type could break interoperability. So the system falls apart without the categorisation schema. 2: For attributes we have another name-space saying that anything starting with "occi." is reserved and any custom attributes MUST start with the reversed domain name of each provider, e.g. "com.example.". Again, if we drop the attribute prefix requirement we can no longer control future additions of attributes in the OCCI spec. If you say that a particular provider must not have conflicting attribute names _today_ he may for example add a "ipv6" attribute. Now if we decide to add a new attribute, e.g. "ipv6", to a future version of OCCI Infrastructure we can never know if there isn't a service provider out there who already have that attribute in use (with different semantics). Therefore we would not be able to add anything and guarantee backwards compatibility. Please feel free to prove me wrong, I would be glad if you did. Now what we _could_ do is to instead of saying "compute": { "memory": 4, ...} we would say: "http://schemas.ogf.org/occi/infrastructure#compute": { "memory": 4, ... } That would work but require more bytes on the wire in most cases. It actually combines the two name-spaces into a single one which is nice though. What say you?
In the category rendering we proposed to use special parameters for different attribute types. e.g. having min_length and max_length for strings and an array of bounds for integers or floats.
Different restrictions for different types, sure. I would just make a slight modification and say, e.g.: "string": { "min_length": 4, "max_length": 32 } That way it is much easier to strictly specify which restrictions apply to which type. Could you make a list of the restrictions applicable to each of the defined attribute types?
Here are some ideas: http://pastebin.com/y2K7bu1S
I do not really see the purpose of the URL list format you call "collection". You already have this with text/uri-list. In the collection example you call "entities" you use a JSON array at the top level. This is a security problem and must be avoided. That is partly why I choose to say { "collection": [ ... ] }. Basically if you always have an json object at the top level silly people doing eval() on the received json data will be kept safe. I see you made the format really lean which is nice. How did you intend to render Link attributes?
I like the "choices" property, only make it a json array instead of a new string format which need to be parsed.
What do you think about them?
Great feedback, keep it coming! :) regards, Ralf
participants (4)
-
alexander.papaspyrou＠tu-dortmund.de -
Feldhaus, Florian -
Gary Mazz -
Ralf Nyren