The value of this architecture is that each layer leverages the social components of the lower layer's architecture.

Read whole article...

The point which David Booth has brought up, not for the first time, and which Pat has expounded very well, that no symbol can ever have completely unambiguous meaning is, yes, quite valid. There are several such points which we have to go over every now and again (preferably out of the critical path of working group work) and agree we all understand it and agree that we can all continue in practice without it. And indeed continue in theory without it as well. And Pat, you have lead us through that journey from philosophical foundationlessness to logical foundations before and maybe you can help us again or just point to where you did before. And Graham you make an important distinction.

There are lots of models, I am sure, one can make of ambiguity and language and communication which will allow us to do this, and they may differ in how they work and it probably is best that we agree they exist but not get hung up arguing about which one is "right". They will all be imperfect, but good enough.

Read rest of article...

The document is written for those who are technically aware or intend soon to be, so it sparse on explanation and heavy in terms of terms.

Read rest of article...

The aim of thes article is to summarize in one place the axioms of Web architecture: those invariant aspects of Web design which are implied or stated in various specifications or in some cases simply part of the folk law of how the Web ought to be used. Especially for these latter cases, this article is designed to tie together the Web community in a common understanding of how we can progress, extend, and evolve the Web protocols. Terms such as "axiom", and "theorem" are used with gay abandon rather than precision as this not a mathematical treatise.

Read rest of article...

Read whole article...

Beneficent Apps

It is a sign of the times (2010's) that we even have to talk about these. Back in the days of floppy disk based PCs, when you would spend your money on a cool program (or App as we say now) you would spend money to be able to do useful, fun things. Play a game. Fly a plane. Write an essay. Do your taxes. You would typically have the program in the A: drive and put a disk for your data in the B: drive.

The data on the B: drive was completely in your control. You could use it with different programs in the A: drive. The program in the A: drive was our tool. It helped you do your work. It worked for you.

Until, that is, for me one day when Quicken, the program I had bought ages ago to do my finances and taxes, when it had done my taxes, asked words to the effect of "Are you sure you have enough insurance? Would you like to buy some great insurance?"

That was the end of an era: the era in which I trusted Quicken to be my representative and work on my behalf. Of course millennials are so used to doing everything on the web, using web-based tools, and so used to those web-based tools actually working in someone else's interest, that they may assume this is normal, and take this as the default.

But in fact in this world we loose something very important. The basic human ability to use a computer provides a wonderful level of empowerment. There is something important about a program which represents me.

While Beneficent Apps are not the norm on the web, or even on mobile devices, they in fact common. Every open source app is (should be!) a beneficent app. These are apps which are developed by a community for its own use, and generally they are developed with the needs and wants of the end user in mind.

Web browsers are important Beneficent Apps. They are crucial as the tool with which the person interacts the web and all the crazy wonderful stuff out there In the HTTP protocol spec, the browser is in fact called the User Agent. Web browses must protect and the serve the user in lots of ways

• Help the user know and understand what party (website owner) they actually talking to
• Help the user remember where they have gone
• Help the user curate a subset of the web which is valuable for them
• Store safely passwords and keys
• Help them avoid being tricked, fooled, or manipulated on line
• and so on

Tips

When designing a Beneficent App, always just think at each design choice -- what would the user want the app to do? If you are thinking of yourself as a main driving user, then think about use cases which empower, and connect to other powerful things you can do or will be able to do. But also think of users who are differ from you in many ways - their level of tech ability, their preferences about being social or not, their situation, their personality.

If you have an income stream from selling the data from your users, then you are not likely to build a Beneficent App by default.

Metrics

How to you measure how good is a beneficent app? It is so easy to make metrics for non-beneficent apps: the engagement level, click-through, the ad revenue or sales revenue they provide. It is more difficult to measure how useful you have been to your user. A user may just be really well informed of something really important, but not do anything which your app could pick up. So yes you can survey them, but now lets's look at measuring their activity. If your app is a something which helps people organize parties, then you can measure the number of parties which people organize. (not to mention whether they were great parties :-).. but those end goals come infrequently, so you could measure the amount of stuff going on to the end of the end goal: the amount of chat (how about sentiment analysis of the chat as to whether it is happy, constructive?) the extent to which the to-do system works - do tasks get done by different people to the people who raise then, for example -- once think of lots of potential things you could measure which may or may not be useful. Then when you have that list, you can look at previous parties and see ho they were involved with making great parties, or with actually getting the party organized at all (which of those would you want to optimize for?) You can also try your apps out, doing A/B testing in the next version as things to optimize for. But also beware of unexpected negative effects. Did they people building social networks imagine the effects on teenage health of a beauty based economy? Probably not, but now we know that systems build to be happy centers of collaboration can end up being toxic for classes of users. To be beneficent, you have to also do no harm!

Rugulating Agents

The concept of apps which work for you, the concept of something which is your agent, is not in fact foreign to our current world. We have it, after all with doctors, and with lawyers. A doctor takes the Hippocratic Oath, or some form of it, which they commit to operate in the interests of the patient. Lawyers also are bound to put the interest of their client first.

There are therefore a lots of laws and regulations out there. to take inspiration from when wording commitments which Apps, or the developers which create them, if anyone wanted to craft regulations ot terms and conditions about Beneficent Apps

Beneficent AI

Hey online Ad system, Who do you work for? When you recommend I eat at a restaurant, is that the best one for, me or a the result of an instant online auction for who ever bids most to you for my custom? Hey, Siri, who do you work for? Hey, Alexa, who to you work for?

Can you even imagine an AI that works for you? I can, and he's called Charlie

References

• IETF Hypertext Transfer Protocol (HTTP/1.1): Message Syntax and Routing

To put it in a nutshell, Conceptual Graphs (CGs) are a logic language used for describing closed worlds of logic. They have traditionally had a strong emphasis on two-dimensional graphical representations, but there are conventional serializations, one "Linear Form" much comparable with N3, and one CG Interchange Format (CGIF) which is more official. With various pros and cons, they are basically as expressive as KIF -- and so in way only have to be webized to a basis for the Semantic Web.

Here I go over a few differences and similarities between CGs and Semantic Web work based on RDF.

I will ignore completely "nonsemantic information" ([1], sec2 ) in this short comparison.

Webizing CGs

Let's take the principles of webizing a language and look at how that applies to CGIF or LF, to imagine a semantic web based on CGIF.

The first thing we clearly have to so is modify the CG syntaxes so that each concept and each relation can be a first class object, by having a URI. The syntax modification is just to allow the characters in a URI to be included, so that an arbitrary concept can be referenced, or an arbitrary relation used. A typical way to map URI space to CG identifiers would be to make URI of a CGIF identifier a concatenation of the URI of the CGIF document, and a hash sign and the local CG identifier -- making the local exsting identifier a fregament identifier in URI terms.

Having mentally webized the language, then the question is how such a semantic web language maps onto say languages. This is simplified by the fact that the CG spec [1] gives a mapping to KIF.

Types and Clases

CG and RDF share concept of type. CGs have the restriction that that the worlds of concepts and types, and that of relationships and relationship types, are disjoint. Therefore, you cannot use a CG to express something about a relation using a relation. If one wanted a true bidirectional mapping, then CGs would have (it seems at first reading) to more or less reify -- to describe at a meta level - an arbitrary RDF graph. However, this would not in my opinion be useful. The designers of CGs intended this disjunction, and so the natural mapping is directly from CG concept types to RDF Classes, and from CG relations to Properties, and from CG Relation Types to RDF Classes which are subclasses of rdf:Property.

The semantic web logic language has to be universal in that it must allow expression of any other language; but it certainly does not force every language to be universal itself.

Centralized Notions in CGs

The CG concept of a knowledge base (KB) contains a few centralized ideas. These are not in fact architectural problem with CGs - they are just engineering decisions which were made without the web scaling requirement. Removing does no damage the CG idea at all.

• The ideal of a closed knowledge base, especially that there is a single catalog of all individuals. A KB contains a hierarchy of types, a hierarchy of relations, and a central catalog of individuals. The hierarchies are no trees, but acyclic graphs, so they do not pose a problem above the fact that they are closed - A KB must
• The fact that a concept is associated wiht a single type. In the semantic web, though the original creator of a Thing may define a type, logically statements made by third parties can equally well make type assertions about a thing, and those statements may be in the form of a rdf:type statement.
• A coreference set has to have a single dominant concept.

Properties and relations

The main difference which stands out at first reading is that RDF properties are always dyadic, while CG relations are monadic.

The RDF base model, and the N3 method of extending it to a logical framework, seem to be supported as a base structure, although the lack of N-ary forms shows up as a mismatch, but the existence of arcs explicitly in the CG model of an N-adic relation suggests a natural mapping back into dyadic RDF when n>2. This just leaves a little tension as the two forms coexist.

The CG world is a bipartite graph - one composed of two relations and concepts, which are disjoint. The RDF world, while it does consist of links which can be thought of as going from thing, via a property, to a thing, does not make properties and things disjoint. Everything is a Thing.

Striking similarities

Some similarities of the CG work and the semantic web to date are striking. Both are inspired largely by circles and arrows diagrams, and in LF and N3 this even shows though in some syntactic forms. People have through the ages been writing circles and arrows on whatever material they had to hand [Enquire, cavewriting] and in N3 I tried to take this very simply into unicode with

w3c:Michael  >- org:member -> w3c:team .

There was a certain feeling of recognition on seeing John Sowa's

[Go]-
   (Agnt)->[Person: John]
   (Dest)->[City: Boston]
   (Inst)->[Bus].

which in N3 would be

@prefix : <#>.
[a :Go]
   >- :agent -> [a :Person; = <#John>];
   >- :dest -> [ a :City; = <#Boston>];
   >- :inst -> [ a :Bus].

remarkable down to the final period. Both syntaxes also have backward arrows a <- (p) <- b in CG's LF, and a<-p-<b in N3. (See also: the same in RDF)

Contexts

The concept of "context" occurs very equivalently in CGs and N3, where in both cases a formula is built using quotation. In N3, the braces were introduced to encapsulate a set of information and talk about it as a set. Using an example from [1], loosely "Tom believes that Mary wants to marry a sailor":

[Person: Tom]<-(Expr)<-[Believe]->(Thme)-
   [Proposition:  [Person: Mary *x]<-(Expr)<-[Want]->(Thme)-
      [Situation:  [?x]<-(Agnt)<-[Marry]->(Thme)->[Sailor] ]].

In N3 this would be, mapping dyadic relations to RDF properties,

<#Tom> a :Person; :believes [a :Proposition; = {
    <#Mary> a :Person; :wants [ a :Situation; = {
        <#Mary> :marriedTo [ a :Sailor ]
    ]}
]}.

(In the above, the "=" is an statement of equivalence which makes up for the inability otherwise of N3 syntax to allow an anonymous context to be subject and object of a statement.) In RDF, my own style is to assume that often the type of a thing, when it can be deduced from the predicate's range or domain, should not be stated explicitly. For example, the object of any believes may be a proposition, and the object of any wants may be a situation. So an N3 expression of the above in practice might be more like:

<#Tom> :believes {
    <#Mary> :wants {
        <#Mary> :marriedTo [ a :Sailor ]
    }
}.

Leaving aside the question of whether this is a good model for the English sentence, and a lot of philosophy and linguistics (which I generally avoid by not trying to express natural language). The CG world often uses diagrams, such as this one from [1] to describe the above formula:

In N3, the circle-and-arrow diagram I would draw would include an arrow from the rectangle for the situation to the [circle] for the marriage to indicate that there is a universal quantification there.

There are other mappings which once could made, none of which give quite such a neat result. One mapping of CGs to RDF would map the CG arcs to RDF properties, which for the above would be:

[ a :Belief;
    :expr <#Tom>;
    :thme: [ a Proposition; = {
        [   a :Want;
            :expr <#Mary>;
            :thme [ a :Situation; = {
                [ a :Marriage; :agent <#Mary>; :thme: [a :Sailor]]
            }
        ]
    }]
].

In English this would be, "There is a belief, experienced by Tom, that "there is a want, felt by Mary, that there should be a situation: ``Mary is married to a Sailor'' ".

Quantifiers and Lambda

I have not gone into the comparison in great detail in this area. Both N3 and CFIF have existential and universal quantification, though the universal quantification is declared an area of the spec under development called "defined quantifiers". Both have, like RDF, implicit existential quantification from anonymous nodes.

A question I did not resolve in CGIF if how one can determine the scope of a quantifier introduced using the "?x" and "*x" terminology. There was a clarification in [1] that (I think) universal quantifiers have a higher scope than existentials of the same scope -- the same convention as in N3. In N3 in the model one has to link the quantified variable directly to its scope context using a log:forAll or log:forSome statement.

N3 has no Lambda as such. Once can write out a double implication define the meaning of a new term (Property or small set of related properties) by giving a double implication with the equivalent formula, using universally quantified variables for the formal parameters.

The issues faced in the two designs do a appear to have a high overlap. The semantic web has to work also in an open context, defining the meaning, if any, of a nested expression when referred to out of context.

Conclusion

Conceptual Graphs are easily integrated with the Semantic Web as it is, the mapping being apparently very straightforward. The export of a CG in CGIF or LF into N3 looks to be a suitable exercise for the reader ;-). An interesting and more challenging exercise would be to build a CG machine -- and a modified CG syntax -- which can import a graph containing URIs which reference external concepts. The problem that relation types in CGs are not concepts is not huge, as there are many systems - especially ontological systems -which have a similar restrictions and with whom interchange would be possible.

There is an interesting subset of CGs, called "simple graph" which are all one context, with no negations or "defined quantifiers", but which can contain universal quantifiers, and these map directly into the RDF M&S 1.0, or N3 without braces.

The RDF base model, and the N3 method of extending it to a logical framework, seem to be supported as a base structure, although the lack of N-ary forms shows up as a mismatch.

All in all, there is a huge overlap, making the two technologies very comparable and hopefully easily interworkable.

with the Semantic Web

Summary

It interesting to use the Semantic Web for connecting the sciences because increasingly major problems can only be solved by using many fields at once; and because scientific information naturally tends to be "data", ie. relational, logical and/or numeric in form, and so Semantic Web technology is easy to apply.

The need

No scientific discipline is as island. The fields of study to which we give names have fuzzy edges, and overlap one another. They are in fact connected in a loose web which evolves with time, as new fields arise, and we change our perceptions of existing ones. Consider physics , physical chemistry, organic chemistry, cell biology, proteomics, genetics, epidemiology, medicine, pharmacology: wheras one might be an expert in one without being an expert in all of them, one typically has to have a knowledge of neighboring fields.

Of the challenges which confront science, many interesting ones, particularly in the study of the human biology, seem to require the tracing of pathways though many fields. In searches for cures for AIDS, for cancer, or for new viruses such as the SARS, the amount of information to be brought to bear is huge, but spans many disciplines.

Now, naturally, different fields have come up with different ways of modelling their data, different standards for recording it. This makes it very difficult to try out new ideas which cross fields: one has to negotiate for the conversion and transfer of data in each case. This is normal. It takes great time and effort to bring more than one group together to use common data formats and common vocabularies.

The solution

The Semantic Web technology is designed specifically to overcome this problem in a decentralized fashion. That is, it is designed to allow conceptual connectivity between neighboring fields to be set up retrospectively and incrementally. Retrospectively, in that often the modelling has already been done in each field and the data already exists. The overlap of concepts only partial, but adding the metadata which expresses that overlap where is does exist is valuable. Incrementally,n that one does not re in that one does not redesign the data models at once, but instead work at the interfaces progressively building links between related concepts.

The Semantic Web language rise above the level of XML, at which document structure is defined, to the level at which the classes of real things in the field in question are defined, the relationships between them and their properties.

Openness

During the early years of the WWW, an element of reluctance was a hesitation by companies to allow information such as their catalogs or parts lists to be available to the general public. This hesitation evaporated when it became clear that only those companies about whose products information was freely available on the web were likely to be involved in any commerce at all. Currently, funders of science have been known to bemoan the disappearance of the original data upon which reports and papers were based. We discovered with the web of human-readable information that much of the benefit was serendipitous: information was used to advantage in ways that its publisher could never have imagined, and the enquirer who started off surfing for a particular solution often finds quite different solutions to that envisaged, not to mention solutions to quite different, but equally pressing problems.

The history of science is peppered with discoveries made serendipitously - from the proverbial bath of Archimedes through the discovery of penicillin, to the discovery of the effect Viagra. If we are to make new discoveries using information on a huge scale, we will need to emulate the openness of the minds of these researchers by making scientific data available in a Semantic Web so that crazy hypotheses can be tested in a few moments harnessing data from many diverse fields.

Indeed, science itself is not an island, as, for example, a epidemiological survey often yields results when joined with geographical an economic data. The search for a disease outbreak could take one into weather patterns, corporate financial statements, or flight timetables. It is important that the scientific Semantic Web is seen as one interoperable part of the larger Semantic Web.

One particular aspect of openness is the lab notebook. The notebook is by tradition a write-only medium in which the scientist writes what he or she did, the environmental conditions a the time, and the results observed. Often such information fades but occasionally it becomes important after the fact. Semantic Web standards, and the use of Semantic Web-aware instrumentation, may make the recording of these incidental things easier. By analogy with the lab notebook, a researcher group may keep a lot of metadata which it may not wish to publish, at the time but which may be useful to posterity. For this information, we need to find a suitable policy which works for everyone involved.

In the longer term, the Semantic Web will by its existence highlight issues such as privacy, the anonymizing of clinical trail information, the protection of possibly security-sensitive infrastructure information, the meaning of copyright especially of compilations, and so on.

Early Steps

Although there is much work yet to do in developing Semantic Web technology, basic standards exist. The Web ontology Language (OWL) allows ontologies to be written so they can be read and processed by machine; the Resource Description Framework (RDF) allows data to be published using OWL ontologies, so that the data itself can be published and re-used by others.

The building of the Semantic Web is a distributed, decentralized task. It behooves those of us who have information which may be useful to others to model it carefully, to discuss ontologies with our neighbors, and expose the information on the web. (It would not be unreasonable to make such publication a condition of funding.) What can be done to encourage this in the early days, to get the snowball rolling?

Firstly, it would be useful to create some simple ontologies for common basic concepts of science. Weights and measures, the periodic table, physical constants, and simply molecules cry out for a standard description. The sort of data would be valuable as a basis for much more complex scientific data, but also would be a great resource for schools. This basic ontology and dataset would also be a service for other fields: one could see chemical data being used as a basis for hazard information, for food and drug information, and for the chemical supply industry, for example.

Secondly, a few example datasets of great general value would demonstrators of how things should be done, and probably give rise to new tools and experience to be passed on. Geophysical, meteorological, pharmaceutical incompatibility information, e many candidates for early adoption.and genome data come to mind, but there must be many candidates for early adoption.

Initiatives to bring scientific data to the Semantic Web could originate in individual researchers, by funding groups, by journals, or scientific associations or academies. If the grow can be compared with can be compared with that of the early WWW, it will occurs wherever an individual person understands the potential long-term global benefit, and so finds a way to put in the short-term effort to make it happen locally.

See also: Paper Trail - presented as a a student project

The basic model of the web is a world of information. Theoretically, a mapping between URIs and representations of the resources they identify, and experientially fro a person a space one can navigate.

Interestingingly, trends at the leading edge of user interface development, and at the semantic web development both point to a world which uses a different model. Human interfaces are moving from screens to conversational mode. The semantic web, while very exciting when viewed as a

Human user interfaces use more and more devices such as speech, gestures and so on, which are not screens. What is special about a screen? A screen with a window system presents a large amount of informatoin at the same time to a person. In practice, more or less everything which a person is concentrting on at one time can be presented in its current state. When the number of pixels on a screen broke through a certain threshold (roughly the 640x320 VGA limit) this led to the development of direct manipulation interface metaphors: folders one could open, and drag and drop. The essential things about this is that the computer is at every instant presenting the current state, whether it or the human is manipulating it. The communication betwen personand machine is in terms of the mutual manipulation of a shared state. The web was intended to extend that form of communication by mutual manipulation of a shared state to remote human-human interaction. While the tools and protocols have their limitations (see UI) much of its effectiveness derived from this model. Because fundamental thing is a shared space of information, one can talk about navigation around within the space, and use all the primaval facilities that the human memory has for navigation.

This is all very well, but it was not always so. When computer terminals had only 24 rows of 80 characters, even when they were addressable, there was a tendency for most jobs to use command line interafaces, for example when manipulating files and directories. The interface was conversational, in that the exchanges were small commands and responses. There was a shared abstract state, but it was imagined in the abstract by the person, and held in some unvisualized form by the computer. This too has itas advantages, in that the imagination of a person can well exceed (on a good day) the capacity of a screen in its ability to hold complex interrelated structures. The interesting thing is that now there is a tednedncy to use many devices which do not have the large screen. The screens on cellphones are currently so small that, while one can scale a web page down and adapt it to a small screen, this might be chosing simply the wrong interface metaphor. When the audio phone only is used, then the shared state becomes zero and the interface is completely conversational again.

The characteristic of a conversation is the state is the set of utterances, or messages, which have been conveyed. This is differenet from a shared expression of a commonly agreed state. The Paper Trail concept links these two modesl in the Semantic Wee Semantic Web, by formally defining the overal agreed state as a function of messages to date. A service which allows a phone user to browse the web converts the other way: it conveys part of the the space of information by means of a conversation. It is is important for a number of reasons.

• It allows us to formalize the models of human-machine interface which are in fact conversational for many non-screen devices;
• It allows us to formalize social, for example commercial, transactions for which the paper trail is in fact th emost accurate model anyway;
• It provides us with tools we can use for formally analysing the infrastructure protocols such as HTTP which with which the information world is actually implemented in practice.
• The standardization of XML protocols has, with XML (and RDF), a richness in terms of marshalling data formats to build on, and, with xml-schemas xforms and rdfs, a richness to draw on in terms of languages for defining valid documents, but has no basis yet for defining with equivalent power the validity (and semantics) of a sequence of interrlated messages which are a protocol.

It is not as though the web today itself perfectly matches the stateless model at all. The moment it was created as a basically stateles system, many web site designers took it as their challenge to get around this model in order to create a conversational interface -- and many still do Our concerns about privacy stem largel;y from the knowledge that our "reading" of documents is in fact done by a series of protocols which leave a trace. The P3P project involves quantifying the information transfer which actually takes place. Our handling of HTML forms is getting more complex, and a form itself, becomes, on many sites, the definition os a protocol - a set of valid sequences of information actions.

@@ - already web privacy concerns come from in fact it being a conversation == there is implict state. A

@@ Reasons for formalizaing protcols a la Paper Trail.: uses concepts of validation and will be able to resuse tools - extends semnatics of documnets to semnatics of conversaions. - Creates a formal basis for defining conversaionsal systems of all kinds, including indirctly human language oriented systems.

@@ Machine-machines and human-human convergence

When a group of people communicate amongst themselves, they develop, to a certain extent, their own language. Sometimes, they pick terms understood by one party, talk enough to develop a shared understanding of the meaning of the the term, and adopt it across the group. Sometimes, as discussion proceeds within the group, meanings are adjusted so that they can be used for new concepts which are created or discovered by the group's activity. Sometimes, a group will deliberately and quite specifically make up a new term, choosing it to be hopefully different from any other word or phrase used before. While this is evident in technical groups, this process also happens in all walks of life, legal and political, as well as social and familial.

The result of this process is a new language, a new strain of language, or just a twist in the use of an existing word. The first and motivating effect of this, large or small, is to enable communication within the group. A greater shared vocabulary broadens the scope of common discussion which can be made without misunderstanding. The second, complementary and inexorable effect of this change is to create a common bond within the group, which at the same time, erects a barrier around the group. In most cases, all of this is unintentional. For every linguistic development which promotes communication within the group, a corresponding step change is made in the difficulty of communicating across the boundary, between inside and outside the group

That which makes a group culture stronger necessarily isolates it from others.

The culture of the group comprises many things, but the common terms and their meaning, and the set of concepts net which interconnects those meanings, are a very significant part.

So it is, then, that a working group will, given a free rein, work in relative isolation for several months, and when they have finished have great difficulty explaining the specification documents to their peers outside the group. Often this will be a surprise to the members of the group. They may see those outside the group as rather slow to understand, and those outside the group might see the group as having a tendency to use jargon, or to misuse jargon. It may be worse: those on one side of the boundary may see those on the other as being stupid, malicious, or even heretical.

An incomplete but essential solution to the problem is for those involved to think about what those on the other side of the boundary are thinking. This is hard work. (It involves, we discover, use of specific parts of the brain! [SaxePowell] ). This is the job, in a conversation, of listening, the stuff of most manuals and self-help books on human communication. In a technical setting it can involve a careful study of the words used in the other's seemingly senseless protestations, to build up logically a conclusion of how those words must be related in the other's mind.

The process of forming a common culture for a large community is, therefore, full of this work of listening to others. It slowly builds a new set of common terms. The work of taking an specification from one group, and though review and discussion, getting it to be th subject of consensus in a wide group, will typically involve reexamining the terms it uses and often changing them as the group itself goes through the process which the individuals, or for that matter smaller groups, within it had already done. The motivation, for technical specifications, is to get wider interoperability of systems. The motivation, for diplomatic and political things, is to get a common decisions, and to reduce global strife.

There is constant tension between the need to get things done quickly with less effort, by working within a small group, and the need to get this wider understanding which takes so much more time.

Now, in practice, life is made up of a fractal tangle of overlapping communities, of overlapping cultures. This means that the tension is ever-present. It also means that there is always a small amount of common language shared by a very large number of people, and always a very large number of concepts local to an individual, and everything in between. In centuries before this one, geography played an important role in constraining groups, and so nested two-dimensional pattern existed.

With the Internet and the Web, we can connect things without the constraint of these nested geographical areas. We can chose to be a member not just of communities such as town, region, state and country, but of specialists in a given field, or people with a particular medical condition, or people concerned about a particular global issue,. world wide. This means that the topology of the communities, and the connectedness by some metrics, may be different and in fact better than before. The topology which emerges depends on the individual choices of many people. But there is a hunch that a fractal distribution, emphasizing all scales, will be important.

The Semantic Web is a technology engineered specifically for this situation. Terms are defined in ontologies (groups of consistent, related terms). Ontologies are defined by communities. A given person is involved in many communities. A given message will mix terms from many ontologies. A given operation only requires consistency between parts of ontologies which are in use for that operation.

This will promote work toward greater harmonization, but it will not predicate the operation upon the establishment of a global ontology of everything. We know that a single huge ontology of everything cannot be done, as it the effort of getting consensus on it becomes unimaginable. We know that stovepipe systems with only local ontologies leave us with communication, and especially the re-use of data, which just does not happen, to our great detriment. And so we engineer specifically for a fractal topology.

References

(There are many books on these topics.)

[SaxePowell] Rebecca Saxe and Lindsey J. Powell, It’s the Thought That Counts: Specific Brain Regions for One Component of Theory of Mind.

2013: The discussion of the good and bad of Digital Rights Management software is wide and furious and has been for many years. It connects to the whole issue of how broken copyright law is and how musicians and film producers should be recompensed for their hard work. In the fervent discussion, very extreme positions have been taken, which has led to the debate becoming acrimonious, to the extent that much more heat than light tends to be available. Here we tease out some separate issues which have become entangled.

• The Open Source Software right to have, modify and distribute a copy of the source code to a program one is running.
• The right to have root [Administrator Privileges] on one's computer, i.e. to be able to completely control what software it runs.
• The right to be able to make a copy of something one is listening to or watching.
• fair use like quotation, parody, etc
• archiving
• use on a different machine
• The right to be able to sell music or video in an encrypted form so it cannot be stolen.

DRM video in HTML5 is a tricky issue. Around 2013 the W3C community was split as to whether it should be allowed. The Electronic Freedom Foundation, and Cory Doctorow the author and blogger were very adamant that HTML should not allow DRM, it represented a step toward big company control of computing platforms. It is impossible to build a DRM machine which has the open source condition that user can change it.

One argument was at the level of tactics, basically, there are companies who will never put their movies on the net without DRM, so basically if we don't put DRM hooks in HTML they will just stick with flash and force you install an app, or a completely closed platform like a set-top box - in other words use a completely locked down platform. So it isn't as though making DRM more difficult in HTML5 will make DRM go away: it will just force users off the web.

So should W3C just say "DRM is evil we should not collaborate with it in any way" and end up driving people to native apps where the whole app is locked down on a locked down platform, or should we open up a slot in an open system to allow a locked-down system to be accessed?

Looking at the philosophical objections to DRM, there is no perfect solution. Everything violates one or more of the rights we want to preserve.

Some people just feel copyright is wrong and so there is a right to make a copy of anything you see or hear. And the business model for musicians if live gigs and donations.

Some people feel that they are the best judge of when they will copy something, as while they do often pay for music etc they feel (a) copyright law has been twisted and applies e.g. to 30 year old movies when it should not, and (b) DRM is too extreme as it prevents normal things like fair use, backups, and typically fails in the future when the DRM support system has changes and all your archive files become unusable.

Some people may feel that DRM is worth the value of having a thriving music industry and film industry. They are not too fussed about the archive issue as they don't really watch movies they have bought a long time ago, or they are too young to have experienced the problem. They haven't answered the problem of getting money to for example bands and singer-songwriters who do not have the blessing of a DRM distribution channel.

Some people may feel that while they don't specifically want to steal movies, they do object to having any bit of computational hardware which they don't have root on.

A related question is, what sort of systems can we build to help people give money to those who e.g. write or perform music, with or without DRM, in an open market, with no 3rd party gatekeeper?

A decade on

So W3C did allow encripted media to played in the browser, by standardiing Encrypted Media Extensions (EME). This allows the web site to get access to one of a small set of DRM gadget on the device -- gadgets which the user has no control of.

Looking back in 2023, there has been a huge amount of streaming on the web and off. EME in HTML has been used massively. A certain notable part of it is user-generated content like YouTube, Vimeo, and TikTok. A huge amount is commercial movies, typically nowadays 4k resolution, and TV series, some short limited set of episodes as a genre competing with full length movies; some going on many series.

There is [still] a constant compettion between web sites an apps. You can follow a link to a video clip on the web, and watch it on the web but Netfllix, Youtube, Apple, etc will alwys try to get you to switch to the app so they have more control of your environment, and can store lots more data on your device.

When you share a video clip in the app, it typically generates a link which will take the recipient back to the web version. You can configure your Operating System so that it will recognize links direct to the app.

As a developer, I can still develop code on my Apple laptop and still install open source code and random apps written by other people on it.

As an artist, there is no way for me to make my own DRM platform if a person wants to protect their material. So they have to find a route to the big platforms. Currenly the big streaming platforms are infamous for returning very little funds to the original artists. I can make my music or video available on my website, and give it away for free, or charge for it without protecting it. Patronage and live gigs and merchendise sales may provide revenue.

The problem of updating and synchronizing data in the Semantic Web motivates an analog to text diffs for RDF graphs. This paper discusses the problem of comparing two RDF graphs, generating a set of differences, and updating a graph from a set of differences. It discusses two forms of difference information, the context-sensitive weak patch, and the context-free strong patch. It gives a proposed update ontology for patch files for RDF, and discusses experience with proof of concept code.

Read whole article...

Various mechanisms of social networks contributed to this dysfunction. These include the abuse of personal data through extensive profiling and tracking, the spread of misinformation and clickbait, and the optimization of content to maximize user engagement at the expense of truth and societal well-being. The consequences of these practices are far-reaching, leading to political polarization, mental health issues, and the erosion of trust in democratic institutions.

The response to these challenges must be multifaceted, involving efforts from tech companies, governments, parents, and activists to mitigate the negative impacts of social networks. Crucial elements include transparency, ethical design, and regulatory measures to create a safer and more humane digital environment. We need a collective effort to harness the positive potential of the internet while addressing its darker aspects.

Read whole article...

Tim BL 3 April 1998

If you think surfing hypertext is cool, that's because you haven't tried writing it. If you have found your bookmarks/favorites have become a more and more important part of your life, that's because you have learned to put up with the simplest form of hypertext editing there is as a compromise. If you are using a really intuitive hypertext editor, then tell me about it.

Hypertext editing

The Web is universal and so should be able to encompass everything across the range from the very rough scribbled idea on the back of a virtual envelope to a beautifully polished work of art.

Somewhere near the "draft" end of the scale is its use a hypertext communal or personal notebook which is very close to a major original use of the Web in 1990. In this mode I can browse over notes made by people in my group, and rapidly contribute new ideas.

I'm editing this now on a pretty intuitive editor. AOLPress is may not be a top of the line pape layout tool but it can do some of the things which my original "WorldWideWeb" program could do. I wouldn't say that either of these programs was the ideal interface, but if you look also at things like KMS and Doug Engelbart's interface, you see that for all the fancy HTML we have nowadays, there is some immediacy we have lost.

Here are some things I would like to be able to do very rapidly. Dan Connolly suggested a click count as a way of measuring the effort, with 10 clicks penalty when you have to think of a filename or anchor ID.

Imagine there's no mode, imagine there's no location

A first assumption, by the way, is that you have modeless interface in which browsing and editing are not separate functions. If to edit a page, you have to switch from browsing mode to editing mode, then you have lost already. If you have had to switch to edit mode, and think of a local filename in which to save the file, then you have lost doubly, If you have had to answer lots of difficult questions about where to save absolute or relative links, you have lost yet again and probably messed up the file already! You should not have to think about "where" things are.

Make a link

In WorldWideWeb, you had to

• Select the target phrase
• Hit "command/M" to mark where you were, (Which generated an anchor with a made up name, and remembered it);
• Switch to the document to contain the link if different;
• select the text to be linked;
• Hit "Command/L" to make the link

In AOLPress, I can do the same thing except the "Mark" function consists of three steps: Press the "anchor" button, hit return to accept the program's suggested anchor name, and then hit the "copy URL" button.

In a drag-and drop world, every window should have an icon for the document it holds which can be dragged to make a link. (Later versions of NeXTStep had this with alt/click on the titlebar).

Make a new linked node - Annotate

In WorldWide Web, this was deliberately easy:

• Select a phrase
• Hit "Command/N". (A new node is created)
• Think of a filename in response to the "SaveAs" dialog box :-(

The new node would be created from a template which could set up to have your signature at the bottom, etc. The original phrase was automatically linked to the new node. The cursor was left ready for you to type in what you'd just thought of.

In a world with PICS servers, then a neat operation is to annotate a page you don't have access to:

• Create a new node somewhere where you have write access
• Create a PICS label with a pointer to it
• Store the PICS label on the label server as a label about the annotated node.

The XML LINK work will allow, we hope, a link to be made into the middle of an existing unwritable document with some hope of reliability.

Here are a few other operations which would be very useful when you really use hypertext as a thinking tool.

Excerpt

Dan is always asking for this and doing it by hand. I have never seen an editor which will do it automatically (though Dan has found some javascript hacks that work pretty well).

• Copy to the clipboard a BLOCKQUOTE with inside it a copy of the selected text, linked back to the original document from which it came. Make the link to an existing anchor in the document if one is there, or else a new one if one can be made, or else the document as a whole failing that.

Insert an image

It's always nice to be able to grab a screen shot or a video frame and insert it into the minutes you are taking of a meeting -- but how many keystrokes does it take?

Introduction

The World Wide Web Consortium was founded in 1994 on the mandate to lead the Evolution of the Web while maintaining its Interoperability as a universal space. "Interoperability" and "Evolvability" were two goals for all W3C technology, and whilst there was a good understanding of what the first meant, it was difficult to define the second in terms of technology.

Since then W3C has had first hand experience of the tension beween these two goals, and has seen the process by which specifications have been advanced, fragmented and later reconverged. This has led to a desire for a technological solution which will allow specifications to evolve with the speed and freedom of many parallel deevlopments, but also such that any message, whether "standard" or not, at least has a well defined meaning.

There have been technologies dubbed "futureproof" for years and years, whether they are languages or backplane busses.  I expect you the reader to share my cynicism when encountering any such claim.  We must work though exactly what we mean: what we expect to be able to do which we could not do before, and how that will make evolution more possible and less painfull.

Free extension

A rule explicit or implcit in all the email-like Internet protocols has always been that if you found a mail header (or something) which you did not understand, you should ignore it. This obviously allows people to add all sorts of records to things in a very free way, and so we can call it the rul of free extension. It has its advatage of rapid prototyping and incremental deployment, and the disadvantage of ambiguity, confusion, and an inability to add a mandatory feature to an existing protocol. I adopeted the rule for HTML when initially designing it - and used it myself all the time, adding elements one by one. This is one way in which HTML was unlike a conventional SGML application, but it allowed the dramatic development of HTML.

The HTML cycle

The development of HTML between 1994 and 1998 took place in a cycle, fuelled by the tension between the competitive urge of companies to outdo each other and the common need for standards for moving forward. The cycle starts simply simply bcause the HTML standard is open and usable by anyone: this means that any engineer, in any company or waiting for a bus can think of new ways to extend HTML, and try them out.

The next phase is that some of these many ideas are tried out in prototypes or products, using the fact free extension rule that any unrecongined extensiosn will be ignored by everything which does not understand them. The result is a drmatic growth in features. Some of these become product differentiators, during which time their originators are loth to discuss the technology with the competition. Some features die in the market and diappear from the products. Those successful features have a fairly short lifetime as product differetiators, as they are soon emulated in some equivalent (though different) feature in competeing products.

After this phase of the cycle, there are three or four ways of doing the same thing, and engineers in each company are forced to spend their time writing three of four different versions of the same thing, and coping with the software architectural problems which arise from the mix of different models. This wastes program size, and confuses users. In the case for example, of the TABLE tag, a browser meeting one in a document had no idea which table extension it was, so the situation could become ambiguous. If the interpretation of the table was important for the safe interpretation ofthe document, the server would never know whether it had been done, as an unaware client would blithely ignore it in any case. This internal software mess resulting from having to implement multiple models also threatens future deevlopment. It turns the stable consistent base for future development into something fragmented and inconsistent: it is difficult to design new features in such an environment.

Now the marketting pressure is off which prevented discussions, and there is a strong call for the engineers to get around the W3C table, and iron out a common way of doing things. As this happens, a system is designed which puts together the best aspects of each other system, plus a few weeks experience, so everyone is in the end happier with the result. The companies all go away making public promises to implement it, even though the engineering staff will be under pressure to add the next feature and startthe next cycle. The result is published as a common specification opene to anyone to implement. And so the cycle starts again.

This is not the way all W3C activities have worked, but it was the particular case with HTML, and it illustrates some of the advantages and disadvantages with the free extenstion rule.

Breaking the cycle

The HTML cycle as a method of arriving at consensus on a document has its drawbacks. By 1998, there were reasons to change the cycle.The work in the W3C, which had started off in 1994 with several years backlog of work, had more or less caught up, and was begining to lead, rather than trail, developments. The work was seen less as fire fighting and more as consolitation. By this time the spec was growing to a size where the principle of modularity was seriously flaunted. Any new developments clearly had to be seperate modules. Already style information had been moved out into the Cascading Style Sheets language, the programming interface work was a seperate Document Object Model activity, and guidelines for accessibility were tackled by a seperate group.

Inthe future it was clear that we needed somehow to set up a modular system which would allow one to add to HTML new standard modules. At the same time, it was clear that with XML available as a manageble version of SGML as a base for anyone to define their own tag sets, there was likely to be a deluge of application-specific and industry-specific XML based languages. The idea of all this happening underthefree extension rule was frightening. Most applications would simply add new tags to HTML. If we continued the process of retrospectively roping into a new bigger standard, the document would grow without limit and become totally unmanageble. The rule of free extesnion was no longer appropriate.

Well defined interfaces

Now let us compare this situation with the way development occus in the world of distributed computing, specifically remote rpocedure call (RPC) and distributed object oriented systems. In these systems, the distributed system (equivalent to the server plus the client for the web) is viewed as a single software system which happens to be spread over several physical machines. [nelson - courier, etc]

The network protocols are defined automatically as a function of the software interfaces which happen to end up being between modules on different machines. Each interface, local or remote, has a well documented structure, and the list of functions (procedures, methods or whatever) and parameters are defined in machine-processable form. As the system is built, the compiler checks that the interfaces required by one module is exactly provided by another module. The interface, in each version of its development, typically has an identifying (typically very long) unique number.

The interface defines the parameters of a remote call, and therefore defines exactly what can occur in a message from one module to another. There is no free extension. If the interface is changed, and a new module made, any module on the other side of the interface will have to be changed too, or you can't build the system.

The great advantage of this is that when the system has been built, you expect it to work. There is no wondering wether a table is being displayed - if you have called the table module, you know exactly what the module is supposed to do, and there is no way the system could be without that module. Given the chaos of the HTML devleopment world, you can imagine that many people were hankering after the well defined interfaces of the distributed computing technology.

With well-defined interfaces, either everything works, or nothing. This was in fact at least formally the case with SGML documents. Each had a document type definition (DTD) refered to at the the top, which defiend in principle exactly what could and could not be in the document. PICS labels were similar in that thet are self-describing: they actually have a URI atthe top which points to a machine-readable description of what can and can't be in athat PICS label. When you see one of these documents, as when you get an RPC mesaage with an interface number on it, you can check whether you understand the interface or not. Another intersting thing you can do, if you don't have a way of processing it, is to look it up in some index and dynamically download the code to process it.

The existence of the Web makes all this much smoother: instead of inventing arbitrary names for inetrfaces, tyou can use a real URI which can be dereferenecd and return the master definition of the interface in real time. The Web can become a decentralised registray of interfaces (languages) and code modules.

The need was clearly for the best of both worlds. One must be able to freely extend a language, but do so with an extension language which is itself well defined. If for example, documents which were HTML 2.0 plus Netscape's version of tables version 2.01 were identified as such, mcuh o the problem of ambiguity would have been resolved, but the rest ofthe world left free to make their own table extensions. This was the goal of the namespaces work in XML.

Modularity in HTML

To be able to use the namespaces work in the extension of HTML, HTML has to transition from being an SGML application (with certain constraints) to being an XML based langauge. This will not only give it a certain ease of parsing, but allow it to build on the modularity introduced by namespaces.

In fact, already in April of 1998 there was a W3C Recommendation for "MathML", defined as as XML langauge and obviously aimed at being usable in the context of an HTML document, but for which there was no defined way to write a combined HTML+MathML document. MathML was already waiting for XML namespaces.

XML namespaces will allow an author (or authoring tool, hopefully) to declare exactly waht set of tags he orshe is using in a document. Later, schemas should allow a browser to decide what to do as a fall back when finding vocabulary which it does not understand.

It is expected that new extensions to HTML be introduced as namespaces, possibly languages in their own right. The intent is that the new languages, where appropriate, will be able to use the existing work on style sheets, such as CSS, and the existing DOM work which defines a programming interface.

Language mixing

Language mixing is an important facility, for HTML, for the evolution of all other Web and application technology. It must allow, in a mixed labguage document, for both langauges to be well defined. A mixed langage document is quiote analogous to a program which makes calls to two runtime libraries, so it is not rocket science. It is not like an RPC message, which in most systems is very strongly ytped froma single rigid definition. (An RPC message can be represented as a structured document but not, in general, vice-versa)

Language mixing is a realtity. Real HTML pages are often HTML with Javascript, or HTML plus CSS, or both. They just aren't declared as such. In real life, many documents are made from multiple vocabularies, only some of which one understands. I don't understand half the information in the tax form - but I know enough to know what applies to me. The invoice is a good example. Many differet coloured copies of the same document used to serve as a packing list, restocking sheet, invoice, and delivery note. Different parts of a company would understand different bits: the financial dividion woul dcheck amounts and signatures, the store would understand the part numbers, and the sales and marketting would define dthe relationship betwene the part numbers and prices.

No longer can the Web tolerate the laxness which HTML and HTTP have been extended. However, it cannot constrain itself to a system as rigid as a classical disributed object oriented system.

The note on namespaces defines some requirements of a language framework which allows new schmata to be developed quite independently, and mixed within one document. This note elaborates on the sorts of things which have to be possible when the evolution occurs.

The Power of schema languages

You may notice than nowhere in the architecture do XML or RDF specify what language the schema should be written in. This is because much of the future power of the system will lie in the power of the schema and related documents, so it isimportant to leave that open as a path for the future. In the short term, yo can think of a schema being written in HTML and english. Indeed, this is enough to tie the significance of documents written in the schema to the law of the land and mkae the document an effective part of serious commercial or other social interaction. You can imagine a schema being in a sort of SGML DTD language which tells a computer program what constraints there are on the structure of documents, but nothing about their meaning. This allows a certain crude validity check to be made on a document but little else.

Now let us imagine further power which we could put into a schema language.

Partial Understanding

A crucial first milestone for the system is partial understanding. Let's use the scenario of an invoice, like the scenario in the "Extensible languages" note. An invoice refers to two schemata: one is a well-known invoice schema and the other a proprietory part number schema. The requirement is that an invoice processing program can process the invoice without needing to understand the part description.

Somehow the program must find out that the invoice is from its point of view just as valid as an invoice with the details fo the part description stripped out.

Optional parts

One possibility is to mark the part description as "optional" on the text. We could imagine a well-known way of doing this. It could be done in the document itself [as usual, using an arbitrary syntax:]

<item>
<partnumber>8137498237</>
<optional>
 <xml:using href="http://aeroco.com/1998/03/sy4" as="A">

   <a:partdesc>
        ...
   <a:partdesc>
 </xml:using>

</opional>
</item>

There are problems with this. One is that we are relying on the invoice schema to define what in invoice is and isn't and what it means. It would be nice if the designer of the invoice could say whether the item should contain a part description of not, or whether it is possible to add things into the item description or not. But in general if there is something to be said we like to allow it to be said anywhere (like metadata). But for the optionalness to be expressed elsewhere would save the writer of every invoice the bother of having to explicitly.

Partial Understanding

The other more fundamental problem is that the notion of "optional" is subjective. We can be more precise about "partial understanding" by saying that the invoice processing system needs to convert the document which contains things it doesn't understand into a document which it does completely understand: a valid invoice. However, another agent may which to convert the same detailed invoice into, say, a delivery note: in this case, quite different information would be "optional".

To be more specific, then, we need to be able to describe a transformation from one document to another which preserves "valididy" in some sense. A simple form of transformation is the removal of sections, but obviously there can be all kinds of level of transformation language ranging from the cudest to theturing complete. Whatever the language, statement that given a document x, that some f(x) can be deduced.

Principle of Least Power

In practice, this suggest that one should leave the actual choice of the transformation language as a flexibility point. However, as with most choices of computer language, the general "principle least power" applies:

(@@justify in greater depth in footnote)

While being able to express a very complex function may feel good, the result will in general be less useful. As Lao-Tse puts it, "Usefulness from what is not there". From the point of view of translation algorithms, one usefulness is for them to be reversible. In the case in which you are trying to prove something (such as access to a web site or financial credibility) you need to be able to derive a document of a given form. The rules you use are the pieces of the web of trust and you are looking for a path through the web of trust. Clearly, one approach is to enumerate all the things which can be deduced from a given document, but it is faster to have an idea of which algorithms to apply. Simple ones have input and output patterns. A deletion rule is a very simple case

s/.*foo.*/\1\2/

This is stream editor languge for "Remove "foo" from any string leaving what was on either side". If this rule is allowed, it means that "foo"is optional. @@@ to be continued

Optional features and Partial Understanding

• Goal: V1 software partially understands V2 document
• Optional features visible as such
• Example: "Mandatory" Internet Draft
• Example: SMIL (P.Rec. 1998/4/9)
• Conversion from unknown language to known language.

Test of Independent Invention

The test of independent invention is a thought experiment which tests one aspect of the quality of a design. When you design something, you make a number of important architectural decisions, such as how many wheels a car has, and that an arch will be used between the pillas of the vault. You make other arbitrary decisions such as the color of the car, the side of the road everyone will drive, whether to open the egg at the big end or the little end.

Suppose it just happens that another group is designing the same sort of thing, tackling the same problem, somewhere else. They are quite unknown to you and you to them, but just suppose that being just as smart as you, they make all the same important archietctural decisions. This you can expect if you believe hat these decisions make logical sense. Imagine that they have the same philosophy: it is largely the philosophy which we are testing. However, imagine that they make all the arbitrary decisions differently. They complement bit 7. They drive on the other other side of the road. They use red buoys on the starbord side, and use 575 lines per screen on their televisions.

Now imagine that the two systems both work (locally), and being usccessful, grow and grow. After a while, they meet. Suddenly you discover each other. Suddenly, people want to work across both systems. They want to connect two road systems, two telephone systems, two networks, two webs. What happens?

I tried originally to make WWW pass the test. Suppose someone had (and it was quite likely) invented a World Wide Web system somewhere else with the same principles. Suppose they called it the Multi Media Mesh ^(tm) and based it on Media Resource Identifiers^(tm), the MultiMedia Transport Protocol^(tm), and a Multi Media Markup Language^(tm). After a few years, the Web and the Mesh meet. What is the damage?

• A huge battle, involving the abandonment of projects, conversion or loss of data?
• Division of the world by a border commission into two separate communities?
• Smooth integration with only incremental effort?

(see also WWW and Unitarian Universalism)

Obviously we are looking for the latter option. Fortunately, we could immediately extend URIs to include "mmtp://" and extend MRIs to include "http:\\". We could make gateways, and on the better browsers immediately configure them to go through a gateway when finding a URI of the new type. The URI space is universal: it covers all addresses of all accessible objects. But it does not have to be the only universal space. Universal, but not unique. We could add MMML as a MIME type. And so on. However, if we required all Web servers to synchronise though one and only one master lock server in Waltdorf, we would have found the Mesh required synchronisation though a master server in Melbourne. It would have failed.

No system completely passes the ToII - it is always some trouble to convert.

Not just a thought experiment

As the Web becomes the basis for many many applications to be build on top of it, the phenomenon of independent invention will recur again and again. We have to build technology so as to make it easy for systems to pass the test, and so survive real life in an evolving world.

If systems cannot pass the TOII, then we can only achieve worldwide interoperability when one original design has originally beaten the others. This can happen if we all sit down together as a worldwide committee and do a "top down"design of the whole thing before we start. This works for a new idea but not for the automation of something which, like pharmacy or trade, has been going on for centuries and is just being represented in the Semantic Web. For example, the library community has had endless trouble trying to agree on a single library card format (MARC record) worldwide.

Another way it can happen is if one system is dropped completely, leading to a complete loss of the effport put into it. When in the late 1980s Europe eventually abandoned its suite of ISO protocols for networking because they just could not interwork with the Internet, a huge amount of work was lost. Many problems, solved in Europe but not in the US (including network addresses of more than 32 bits) had to be solved again on the Internet at great cost. Sweden actually changed from driving on the left to driving on the right. All over the world, people have changed word processor formats again and again but only at the cost of losing access to huge amounts of legacy information. The test of independent invention is not just a thought experiment, it is happening all the time.

From philosophy to requirement

So now let us get more specific about what we really need in the underlying technology of the Semantic Web to allow systems in the future to pass the test of independent invention.

We will be smarter

Our first assumption is that we will be smarter in the future. This means that we will produce better systems. We will want to move on from version 1 to version 2, from version n to version n+1.

What happens now? A group of people use version 4 of a word process and share some documents. One touches a document using a new version 5 of the same program. Oen of the other people tries to load it using version 4 of the software. The version 4 program reads the file, and find it is a version5 file. It declares that there is no way it can read the file,as it was produced in the future, and there is no way it can predict the future to know how to read a version 5 file. A flag day occurs: everyone in the group has to upgrade immediately - and often they never even planned to.

So the first requirement is for a version 4 program to be able to read a version 5 file. Of course there will be some features in version 5 that the version 4 program will not be able to understand. But most of the time, we actually find that what we want to achieve can be done by partial understanding - understanding those parts of the document which correspond to functions which exist in version 4. But even though we know partial understanding would be acceptable, with most systems we don't know how to do even that.

We are not the smartest

The philosophical assumption that we may not be smarter than everyone else (a huge step for some!) leads us to realise that others will have gret ideas too, and will independently invent the same things. It forces us to consider the test of independent invention.

The requirement for the system to pass the ToII is for one program which we write to be able to read somehow (partially if not totally) data written by the program written by the other folks. This simple operation is the key to decentralised evolution of our technology, and to the whole future of the Web.

So we have deduced two requirements for the system from our simple philosophical assumptions:

• We will be smarter in the future
  • Technology: Moving Version 1 to Version 2
• We are not smarter than everyone else
  • Decentralized evolution
  • Technology: Moving between parallel Version A and Version B

The story so far

We are we with the requirements for evolvability so far? We are looking for a tecnology which has free but well defined extension. We want to do it by allowing documents to use mixed vocabularies. We have already found out (from PICS work for example) that we need to be abl eto know whether extension vocabulary is mandatory or can be ignored. We want to use the Web for any registry, rather than any central point. The technology has to be allow an application to be able to convert the output of a future version of itself, or the output of an equivalent program written indpendently, into something it can process, just by looking up schema information.

Evolution of data

Now let us look at the world of data on the Web, the Semantic Web, which I expect we expect to become a new force in the next few years. By "data" as opposed to "documents", I am talking about information on the Web in a form specifically to aid automated processing rather than human browsing. "Data" is characterised by infomation with a well defined strcuture, where the atomic parts have wel ldefined types, such as numbers and choices from finite sets. "Data", as in a relational database, normally has well defined meaning which has rarely been written down. When someone creates a new databse, they have to give the data type of each column, but don't have to explain what the field name actually means in any way. So there is a well defined semantics but not one which can be accessed. In fact, the only time you tells the machine anything about the semantics is when you define which two columns of different tables are equivalent in some way, so that they can be used for example as the basis for joining the two databases. (That the meaning of data is only defined relative to the meaning of other data is of course quite normal - we don't expect machines to have any built in understanding of what "zip code" might mean apart from where you can read it and write it and what you can compare it with). Notice that what happens with real databases is that they are defined by users one day, and they evolve. They are rarely the result of a committee sitting down and deciding on a set of concepts to use across a company or an industry, and then designing the data schema. The schema is craeted on the fly by the user.

We can distinguish two ways in which tha word "schema" has been used:

I will use it for the first only. In fact, a syntactic schema dedfines a class of document, and often is accompanied by human documentation which provides some rough semantics.

There is a huge amount ("legacy" would unfairly suggest obsolescence) of data in relational databases. A certain amount of it is being exported onto the web as virtual hypertext. There are many applications which allow one to make hypertext views of difeferent aspects of a database, so that each server request is met by performing adatabse query, and then formatting the result as a report in HTML, with appropriate style and decoration.

Data about data: Metadata

Information about information is interesting in two ways. Firstly, it is interesting because the Web society desperately needs it to be able to manage social aspects of information such as endorsement (PICS labels, etc), ownership and access rights to information, privacy policies (P3P, etc), structuring and cataloguing information and a hundred otehr uses which I will not try to ennumerate. This first aspect is discussed elsewhere. (See Metadata architecture about general treatment of metadata and labels, and the Technology and Society domain for overveiw of many of the social drivers and related projects and technology)

The second interest in metadata is that it is data. If we are looking for a language for putting data onto the Web, in a machine understandable way, then metadata happens to be a first application area. Also, because metadat ais fundamental to most data on eth web, it is the focus of W3C effort, while many other forms of data are regarded as applications rather than core Web archietcure, and so are not.

Publishing data on the web

Suppose for example that you run a server which provides online stock prices. Your application which today provides fancy web pages with a company's data in text and graphs (as GIFs) could tomorrow produce the same page as XML data, in tabular form, for machine access. The same page could even be produced at the same URL in two formats using content negotiation, or you could have a typed link between the machine-understandable and person-understandable versions.

The XML version contains at the top (or soemewhere) a pointer to a schema document. This poiner makes the document "self-describing". It is this pointer which is the key to any machine "understanding" of the page. By making the schema a first class object, in other words by giving its URL and nothing else, we are leaving the dooropen to many possibilities. Now it is time to look at the various sorts of schema document which it could point to.

Levels of schema language

Computer languags can be classified into various types, with various capabilities, and the sort we chose for the schema document, and information we allow the schema fundamentally affects not just what the semantic web can be but, more importantly, how it can grow.

The schema document can, broadly, be one of the following:

1. Notional only: imaginary, non-existent but named.
2. Human readable
3. Machine-understandable and defining structure
4. Machine-understandable and slo which are optional parts
5. A Turing-complete recipe for conversion into othr langauges
6. A logical model of document

We'll go over the pros and cons of each, because none of these should be overlooked, but some are often way better than others.

Schema 1: URI only

• No supporting documentation
• Allows compatibility yes/no test

This may sound like a silly trivial example, but like many trival examples, it is not silly. If you just name your schema somewhere in URI space, then you have identified it. This deosn't offer a lot of help to anyone to find any documentation online, but one fundamental function is possible. Anyone can check compatability: They can compare the schema against a list of schemata they do understand, and return yes or no.

In fact, they can also se an idnex to look up information about the schema, including ifnromation about suitable software to download to add understanding of the document. In fact this level is the level which many RPC systems use: the interface is given a unique but otherwise random number which cannot be dereferenced directly.

So this is the level of machine-understanding typical of distributed ocmputing systems and should not be underestimated. There are lot sof parts of URI space you can use for this: yo might own some http: space (but never actually serve the document at that point) , but if you don't, you can always generate a URI in a mid: ro cid: space or if desperate in one of the hash spaces.

Schema option 2: Human readable

The next step up from just using the Schema identifier as a document tyope identifier is to make that URI one which will dereference to a human-readable document. If you're a computer, big deal. But as well as allowing a strict compatiability test (test for equality of the schema URI), this also allows human beings to get involed if ther is any argument as to what a document means. This can be signifiant! For example, the schema could point to a complete technical spec which is crammed with legalese about what the document does and does not imply and commit to. At the end of the day, all machine-understandable descriptions of documents are all very well, but until the day that they bootstrap themselves into legality, they must all in the end be defined in terms of human-readable legalese to have social effect. Human legalese is the schema language of our society. This is level 2.

Schema option 3: Define structure

Now we move into the meat of the schema system when we start to discuss schema documents which are machine readable. now we are satrting to enable some machine understanding and automatic processing of document types which have not been pre-programmed by people. Ça commence.

The next level we conside is that when your brower (agent, whatever) dereferences the namespace URI, it find a schema which defines the structure of the document. this is a bit like an SGML Doctument type Definition (DTD). It allows you to do everything which the levels 1 and 2 allowed, if it has sufficient comments in it to allow human arguments to be settled.

In addition, a system which has a way of defineing structure allows everyone to have one and only one parser to handle all manner of documents. Any document coming across the threshold can be parse into a tree.

More than that, it allows a document o be validated against allowed strctures. If a memeo contains two subject fields, it is not valid. Tjis is one fo the principal uses of DTDs in SGML.

In some cases, there maybe another spin-off. You canimagine that if the schema document lists the allwoed structrue of the document, and the types (and maybe names) of each element, then this would allow an agent to construct on the fly a graphic user interafce for editing such a document. This was theintent with PICS rating systems: at least, a parent coming across a new rating system would be be given a ahuman-readable descriptoin of the various parameters and would be able to select

Schema option 4: Structure + Optional flags

The "optional" flag is a term I use here for a common crucial step which can make the difference between chaos and smooth evolution. All you need to do is to mark in the schema of a new version of the language which elements of the langauge can be ignored if you don't understand them. This simple step allows a processor which handled the old language, giventhe schema of the new langauge, to filter it so as to produce a document it can legitimately understand.

Now we have a technology which ahs all the benefits to date, plus it can handle that elusive version 2 to version 1 conversion problem!

Schema option 5: Turning complete language

Always in langauges there is the balance between the declarative limited langauge, whose foprmulae can be easily manipulated, and the powerful programming language whose programs cannot be analyzed in general, but which have to be left to run to see what they do. Each end of the spectrum has its benefits. In describing a lanuage in terms of another, one way is to provide a black box program, say in Java or Javascript, which will convert from one to the other.

Filters written in turing-complete languages generally have to be trusted, as you can't see what rules they are based on by looking at them. But they can do weird and wonderful things. (They can also crash and loop forever of course!).

A good language for conversion from one XML-based language to another is XSL. It lstarted off as a template-like system for building one document from another (and can be very simple) but is in fact Turning-complete.

When you do publish a program to convert language A to language B, then anyone who trusts it has that capability. A disadvantage is that they never know how it works. You can't deduce things about the individual components of the languages. You can't therefore infer much indirectly about relationships to other languages. The only way such a filter can be used is to get whatever you have into language A and then put it though the filter. This might be useful. But it isn't as fascinating as the option of blowing language A open.

Schema option 6: Expose logic of document

What is fundamentally more exciting is to write down as explicitly as posible wahteth new language means. Sorry, let me take that back, in case you think that I am talking about some absulte meaning of meaning. If you know me, I am not. All I mean is that we write in a machine-processable logical way the equivalences and conversions which are possible in and out of language A from other languages. And other languages.

A specific case of course, is when we document the relationship betwen version 2 and version 1. The schema document for version 2 could explain that all the terms are synonyms, except for some new terms which can be converted to nothing (ie are optional) and some which affect the meaning of the document completely and so if you don't understand them you are stuck.

In a more general case, take a language like iCalendar in RDF (were it in RDF), which is for describing events as would be in a personal organizer. A schema for the language might declare equivalences betwen a calendar's concept of group MEMBER ship and an access control system's concept of group membership; it might declare the equivalence of eth concept of LOCATION to be the text description of a Geographical Information Systems standard's location, and it may declare an INDIVIDUAL to be a superset of the HR department's concept of employee. These bits of information of the stuff of the semantic web, as they allow inference to stretch across the gloabe and conclude things which we knew as whole but no one person knew. This is what RDF and the Semnatic Web logic built on top of it is all about.

Contents

Extensible languages

• Introduction
• Requirements
  • Glossary
  • Mixing vocabularies
  • Scenario
  • Local scope
  • Lack of ambiguity
  • Evolving new scheme languages
  • Correctness of documents with multiple vocabularies
  • Granularity
  • Incorporation into the language
• Related resources

Information is powerful stuff.  The world has been enthralled by the power which the Web, the universe of accessible information, gives to people and to groups working and playing together.

Information about information is powerful not just as information, but because it allows one to leverage one's use of information, to benefit from that which is relevant, accurate, stylish, unbiased, or timely, -- whatever one regards as being of "quality" -- without being enmired in that which is not.

Powerful tools are often usable for constructive or destructive purposes just as paper and ink be used for truth or lies, and metal for ploughshares or swords. The Web's power stems from its universality - for example that a hypertext link an point to any information out there, not just a subset.  People have asked whether I regret that the Web has been used for some uses, but I have to reply that if somehow it had been built to control the material which was placed in it, then that would be the technology controlling society, rather than the other way around as it should be.

True, one could take the view that our society is not strong enough to be trusted with a powerful information system.  One could take the view that society does not currently have the wherewithal to prevent the Web from being abused by destructive forces to an extent that the overall pain is greater than the gain.  I do not believe this is true. In the western developed world, at least, I believe that the democratic process will have sufficient control over governments and the judicial process sufficient control of criminals, to continue to defend the health of the evolving society.

We should be very careful, by constant inspection, to ensure that this continues to be the case.

Filtering and Censorship

One of the threats which posed itself in 1994 was of government censorship over information on the Web.  In general, there are information acts which societies regard as legal, and those which are illegal (such as fraud).  The problem which arose was that in the very subjective question of what information is deemed suitable for children, there was a threat that, in order to "protect" children, seeing no other alternative, governments were contemplating making draconian legislation for example prohibiting the transmission of "indecent" material. The problems here were many.

First of all, the concept of "indecent" was being enforced as a central single concept, quite against the distributed subjective nature of its definition in society.  The Web works as a decentralized system, with no hierarchical or other structure to force society into a shape imposed by technology.  This works.  Centralization of such an idea would [prevent the Web from being an accurate mirror of society itself.

Secondly, the problem being solved was the reading of such information by children, not its transmission. Thirdly, the question of "transmission" seemed to include intermediate parties who were not responsible for the content in an editorial or authorship sense. And one could list other problems, but this is enough for the present.

Information Quality

The basic problem being addressed was that of subjective information "quality".  This is the same problem reported by newcomers to teh web who find (typically after a search engine search) too much "junk".

It is unreasonable to ask for information delivered from the web to be of consistently high "quality" if you can't define what "quality" is.  There is a need, then, to be able to represent "quality" in a completely subjective way.

This is what the PICS project was all about.  PICS was specifically aimed at demonstrating that individuals could obtain their own subjective notion of quality without the government having to try to "protect" them by enforcing some centralized notion.  Politically, PICS is a system necessary for the preservation of free speech on the Internet.

The system needed a few different sorts of documents

a "rating system"

which defines a scale or scales on which one might judge a document.  The fact that anyone can create one of these is a strong force allowing decentralization of concept, breaking the problem of the global, centralized definition of for example "indecency".  PICS allows communities of any size (from one up) to establish their own criteria.  Agreement over a large community enhances global harmony, but threatens diversity. Agreement over a small community does the reverse. So in fact some balance is necessary

a "label"

which is a statement about something in terms of the schema.  This can be made by any party, not just author or reader, and certainly not just central government.  These can be created and exchanged in all manner of ways, so the PICS standard for interoperability is essential.

A "profile"

which describes for a given person the particular rating systems and levels on those scales which represent "quality" information at a given time and in a given context.  This sort of information can either be input by a person using a graphic interface (such as a set of sliders in a dialog box), or can simply be transferred from someone they trust, whether family, organization, or friend. Inability to transfer this would prevent people from building their own communities with common standards of trust: hence the importance for this (picsrules) as a standard.

These are all subsets of a general metadata language, designed to be easy for people to use.   In particular, by being limited in their power, they allow graphic interfaces to be built.

On social responsibility of technologists

The argument has been made that PICS technology should be suppressed as the power it gives may be abused by governments.  (There are even those who have suggested that the whole scheme is a government inspired plot to promote censorship and limit free speech.  This is certainly not the case, as neither in the idea,  the funding nor the intent.)  Whereas most readers may find this far fetched, it is worth a response on principle.

As I pointed out when closing the first International World Wide Web conference, speaking to (then a mere 350) geeky web enthusiasts, I firmly believe it is the task of scientists and technologists to be aware of and responsible for the social implications of their work.  This cannot just be left to "professional socially responsible people", as each engineer and scientist is often  best aware of the potential of the work.  Uttered in the auditorium at CERN, whose particle physicists trace their roots through nuclear physics, I don't think the message went unheard, even though it may have sounded strange in such a new field.  Now, (1997) the World Wide Web Consortium has one of its three domains dedicated to the relationship between Technology and Society.

So what about PICS?

The question basically is whether the potential danger of the technology outweighs the freedom and positive good it accords.  You can certainly argue this for nuclear fission, and you can certainly argue it for the wide distribution of firearms in populous countries. Can you argue this for PICS and metadata?  Is there anything about PICS specifically or metadata in general which makes it more of a danger than a boon?

The specific types of document in PICS are very general. As a system, it is quite generalist, and extremely decentralized.  It does as good a job as it can of leaving policy up to others to set, although it does (compared with other systems one could imagine) tend to favor by its nature cultural diversity, and freedom of speech, including freedom to endorse other's work.

 The specifications of communication protocols enable, but do not enforce, what manufactured software will or will not be able to do.  One cannot, therefore, at this level say what individuals will be able to do.  The technology can leave the policy up to others, which leaves other groups to ensure that the values which they hold dear are not lost in new legislation, industry practices, or public apathy.

A metainformation system allows one to talk about information. It enables all kinds of uses of information

• finding information
• talking about information
• making laws about information
• breaking laws about information

It is not the place of a technical metadata system to try to limit the statements one can make with metadata, or the laws if any which are made.  That is the role of the democratic process and whatever government the people trusts. The W3C as an industry consortium can act for industry in promoting standards, but cannot act to create laws.  What we can do is explain to lawmakers and others the effect and intention of technology.  That is what this article attempts to do.

Conclusion

So Metadata, PICS and otherwise, is powerful, as is information in general.  Constant vigilance by concerned members of the public, industry and government is a very important part of the system of controls which keeps society healthy.  The PICS technology was created specifically in order reduce the risk of government censorship in civilized countries. It was the result of members of the industrial community being concerned about the behaviour of government. The indications are that in this it will succeed,  but that does not remove the need for such vigilance.  

To conclude, out of fear or ignorance, that PICS is more of a danger than it is a boon would be throw the baby out with the bathwater.  Metadata is not just a new tool, it is the start of a machine-understandable web (a "web phase 2") of information whose impact should be as empowering  to humanity as the human-understandable web of today.  We must understand it as we build it.

Ontology languages

The RDF schema languages, RDF Schema and OWL, tell you implications one can draw from RDF Model data. They also tell you what things do not make logical sense. Therefore in a sense they indirectly have the function of constraining what RDF data one can write, though just by telling what would be nonsense (false). So they can in a rather weak way be used to guide a user interface. But that won't do what we need.

Other schema systems, like that of schema.org, give suggestions as to what predicates can be used to talk about objects of a given class. That is useful, but still is not enough.

In this document, we will discuss three kinds of technologies to help with building apps on top of data:

1. Shapes explain to machines what data should look like, independently of how that data is displayed to a user.
2. Forms are a user interface allowing people to read and write data in a specific shape.
3. Footprints explain to machines where new data should be stored.

Read rest of article...

This article was originally entitled "The Fractal nature of the web". Since then, i have been assured that while many people seem to use fractal to refer to a Zipf (1/f) distribution, it should really only be used in spaces of finite dimension, like the two-dimensional planes of MandelBrot sets. The correct term for the Web, then, is scale-free.

This isn't an observation so much as a requirement.

I have discussed elsewhere how we must avoid the two opposite social deaths of a global monoculture and a set of isolated cults, and how the fractal patterns found in nature seem to present themselves as a good compromise. It seems that the compromise between stability and diversity is served by there the same amount of structure at all scales. I have no mathematical theory to demonstrate that this is an optimization of some metric for the resilience of society and its effectiveness as an organism, nor have I even that metric. (Mail me if you do!)

However, it seems from experience that groups are stable when they have a set of peers, when they have a substructure. Neither the set of peers nor the substructure must involve huge numbers, as groups cannot "scale", that is, work effectively with a very large number of liaisons with peers, or when composed as a set of a very large number of parts. If this is the case then by induction there must be a continuum of group sizes from the vary largest to the very smallest.

This seems to be a general rule which can guide our design, and against which we can measure actual patterns of use.

It is in fact another aspect of the tension between many languages and one global language. Locally defined languages are easy to create, needing local consensus about meaning: only a limited number of people have to share a mental pattern of relationships which define the meaning. However, global languages are so much more effective at communication, reaching the parts that local languages cannot. This tension is exemplified in the standards process, when ideas have to be exposed to successively larger and larger groups, with friction and hard work at each stage.

Other interesting things to model passing though a fractal system include DNA traits in intermarrying populations Someone suggested (who?) that the invention of the bicycle made a great difference to average health in the Welsh valleys because it allowed greater intermarrying and so increased the effective gene pool size Clearly, global travel could end up reducing the diversity. viruses propagating through schools and traveling business people; and problems propagating to someone who has a solution are more good exercises (State your assumptions!).

Zipf happens

Whether we like it or not, early measurements of web traffic by the DEC WRL firewall showed DEC employees browsing sites with a Zipf (1/n) distribution of popularity. (Anyone got any other measurements? [Neilsen 1997]). Recent analyses suggest the Web becoming smaller for its size seem to use.

How can we use knowledge of the Web's fractal nature? By planning network bandwidth between long-range and short-range communication, planning for cache usage, etc. The physical network can be expected to have a variety of scale geographically, like the road system. However, the structure of the Web is interestingly different because of the lack of two-dimensional constraint. The challenge is to use this flexibility in building an effective society on top of the Web.

Looking for a metric

What do we mean by "effective"? We mean we would like to combine scientist's creative ability and knowledge to find a cure for AIDS. We would like to preserve world peace by allowing xenophobia to disperse in a web of understanding, while at the same time preserving the diversity of culture which gives the human race its richness. These are of course the same classic problems of the management of a large organization, of combining individual creativity with corporate vision.

If the web of society has an imbalance, we pay for it. We pay for insufficient global understanding with war. We pay for insufficient family communication with broken families and unsupported individuals. At any level of scale, missing social structure at that scale will prevent problems at that scale being addressed, and also prevent resources at that scale being used. It would therefore be great to have a way of measuring for a given web the degree to which it has a balanced fractal pattern, and if not where its weaknesses are.

Those looking for the "small world" effect chose metrics such as the maximum or mean value of the shortest path between any two points. This gives us a metric for effectiveness at the global scale, but not of the chewiness.

Clustering algorithms can produce globs of various sizes, and a measure of the chewiness of a web may be that the cluster sizes have a Zipf distribution. For example, using Jon Kleinberg's algorithm (which for a link matrix A associates concepts with the eigenvectors of A*A), the strength of the cluster is the value of the eigenvalue, and (while this does not directly indicate size) an interesting test would be on the relative absolute values (squares?) of successive eigenvalues.

Looking it at from the point of view of an individual (a graph node), an interesting question is the proportion of the traffic which is to local or more distant nodes. In Marchiori's model [Marchiori] traffic flows between two nodes in inverse proportion to the resistance of the shortest path. The total "efficiency" is deemed to be the total flow between all pairs of nodes. Can we measure a "chewiness" which measures the approximation of the system to a fractal distribution of long and short range communication? If the Marchiori model were modified to use parallel conductance (more like a real signal flow system) then would this be simpler?

Suppose for example we look at the amount of connection we have with nodes whose distance, or groups whose size, is of each order of magnitude and look for smoothness up to the global level.

Stop Press

2000/03

Well, here I was thinking that while it is intuitively clear that society has to be fractal, I didn't know a mathematical justification for it, when Jon Kleinberg comes up with what for me is his second cool web result.

This is a paper takes the case of a two-dimensional grid. It imagines each cell having a certain distribution of links of various lengths. It demonstrates that in order to achieve the connectivity a la 6 degrees of separation which scales with the log of the size of the system, then the distribution of link density as a function of distance must be precisely an inverse-square law. That is, each cell must have the same number of links (on average) to cells 1-10 squares away as to cells 10-100 away, etc. Anything more local or more global leads to less of a small-world phenomenon: this is the only scalable solution.

True, this applies to a geographical grid, and a square rather uniform one at that. However, He does generalize it to more dimensions. Furthermore, you can see logically how the system works. To get a postcard to an arbitrary person in Massachusetts through a network of friends, you must have enough local friends to be able to find someone who will know someone in Massachusetts. The person they find in Massachusetts must be able to pass it to people successively closer and closer to the target. this only works if there is connectivity on each scale. True, no one has derived the metric of the number of hops a message takes as being an essential metric for systems, but on the other hand there is a clear analogy with the number of hops between a problem and a solution in a large organization .

Other work:

• Living semantic web

Data from Swoogle April 2005

• All these series follows Zipf's distribution, except the tail
• The sharp decrease the tail in "class populated" shows that the most populated classes highly correlated such that their are populated by almost the same amount of SWDs. Similar situation can be observed in other series.
• The closeness of the sharp decrease of "class populated" and "property populated" is caused by the co-existence of certain classes and certain properties.

Postscript - A personal exercise

There will I am sure be a lot of ways in which the fractal requirement is used in web design. You can also use it in that task of figuring out how you fit in to society at large (and at small). Do your personal interactions spread across the scales? Here is a self-help chart to help think about this. You fill in the groups in your life.

Another way to do this is find 11 jars, and label one with each scale in powers of 10. (You don't have to paint them but it helps).

Put marbles in each can for each time period you spend on matters at a given scale, such as an international meeting, or a school sportsfield, or with your family, or alone in a treehouse. How well balanced do the jars become?

As a social person, do you spend enough time with groups of each size? If not, are there people one click from you who do, and through whom you are indirectly present in those groups? One of the concerns is that the last column - the global column - tends in my observation to get the smallest amount money at least, as in the US federal and state and town taxes are spread around the other areas but the level of international aid is very much lower. The cool thing is that I think people are born with DNA which gives them a healthy interest at all these levels. People who stick at one scale all their lives feel very uncomfortable. Maybe our preferences have evolved to form naturally a fractal society.

Total Cost of Ontologies (2005)

One of the interesting things about assuming a fractal distribution is you can think about the number of ontologies an the time it takes to make them, and the total cost of using ontologies. So let us for example naivel assume that
ontologies are evenly spread across orders of magnitude; committe  size goes  as log(community), time as comitee^2, cost is shared across community.

Q: How can the semantic web work...

... when we are all in one big domain of discourse but people are all making their own local ontologies? (2007/3/3)

Rather than 'domain of discourse' , or set of things considered, I think of 'community', set of agents communicating using certain terms. When one thinks in terms of domain of discourse, one tends to conclude that everyone who talk at all about a car (say) has cars in their domain of discourse and so everyone must share the model which includes the single class Car.

It isn't like that though. An agent plays a role in many different overlapping communities. When I tag a photo as being of my car, or I agree to use my car in a car pool, or when I register the car with the Registry of Motor Vehicles, I probably use different ontologies. There is some finite effort it would take to integrate the ontologies, to establish some OWL (or rules, etc) to link them.

• Everyone is encouraged to reuse other people's classes and properties to the greatest extent they can.
• Some ontologies will already exist and by publicly shred by many, such as ical:dtstart, geo:longitude, etc. This is the single global community.
• Some ontologies will be established by smaller communities of many sizes.

Why do I think the structure should be will be fractal? Clearly there will be many more small communities, local ontologies, than global ones. Why a 1/f distribution? Well, it seems to occur in many systems including the web, and may be optimal for some problems. That we should design for a fractal distribution of ontologies is a hunch. But it does solve the issue you raise. Some aspects of the web have been shown to be fractal already.

• - The connections between the ontologies may be made after their creation, not necessarily involving the original ontology designers.
• - There is a cost of connecting ontologies, figuring out how they connect, which people will pay when and only when they need the benefit of extra interoperability.
• - Sometimes when connecting ontologies, it is so awkward there is pressure to change the terms that one community uses to fit in better with the other community. Again, a finite cost to make the change, against a benefit or more interop.

Yes, if web-based means an overlapping set of many ontologies in a fractal distribution. In his fractal tangle, there wil be several recurring patterns at different scales. One pattern is a local integration within (say) an enterprise, which starts point-point (problems scale as n^2) and then shifts with EIA to a hub-and-spoke as you say, where the effort scales as N. Then the hub is converted to use RDF, and that means the hub then plugs into a external bus, as it connects to shared ontologies.

So the idea is that in any one message, some of the terms will be from a global ontology, some from subdomains. The amount of data which can be reused by another agent will depend on how many communities they have in common, how many ontologies they share.

In other words, one global ontology is not a solution to the problem, and a local subdomain is not a solution either. But if each agent has uses a mix of a few ontologies of different scale, that is forms a global solution to the problem.

Conjecture

The conjecture is that there is some model which reasonably well described these systems, and that given that model one can show that the scale-free distribution of communities is optimal.

There are many other questions. Of course existing systems on the earth may be very much influenced by the geographical reality of a two-dimensional surface. Historical groups have been nested geographically. So though there may be aspects in which community size is scale-free, that maybe a completely different optimisation problem from the one we have when on the Internet anyone can connect to anyone. If you could devise an algorithm for connecting people into groups, and so that they each participated in communities of different sizes in a scale-free way, then how much more effective (at solving problems, etc) can you make a web-based society which ignores geographical borders? To what extent does humanity as currently connected by the web in fact deviate from geographical nesting anyway?

The URI by itself is a powerful thing, but there is a more powerful concept which is the URI reference.

The URI reference is a thing you build by taking a URI for an information object, adding a "#" sign and then a Fragement identifier. (The last term is historical, so try not to thinl of it necessarily identifying a fragment).

The fragment identifier is a string after URI, after the hash, which identifies something specific as a function of the document. For a user interface Web document such as HTML poage, it typically identifies a part or view. For example in the object

         http://foo/bar#frag
 

the string "frag" is the fragment identifier. It is badly named, as it can identify anything.

(Depending on where you look, the URI is considered to include the fragment identifier, or to have the fragment identifier appended to it.  This is important for the BNF, but in practice you will find people using the terms URI and URL loosely to things which do or do not include a possible fragment identifier. Formally, the URI does include the fragment ID)

In practice, you can divide the processing which occurs when following a link using HTTP into three steps:

1. The client figures out which server to contact by parsing part of the URL, and sends the URL as a request to the server;
2. The server figures out which object is referred to by parsing the rest of the URL, and returns some rendition of it to the client;
3. The client presents all or part of the object to the user

The last part typically involves finding some software class which can handle the given MIME type, and passing it the data stream.  At the same time, the fragment identifier is passed as a parameter to the created object.

For HTML, the fragment ID is an SGML ID of an element within the HTML object. For XML, if it is just a word, then it is the XML ID of an element in the document.

Axiom

The significance of the fragment identifier is a function of the MIME type of the object

This means that the fragment id is opaque for the rest of the client code.  The HTTP engine cannot make any assumptions about it.  The server is not even given it.

It also means that for any new data type one can be creative about using the fragment ID in a relevant way. For example, for a 3D object the fragment ID  could give a viewport. For a music object, the Fragment ID could give a  section in time, or a set of parts, or it could include a suggested tempo.  For future versions of HTML, the fragment ID could be made more powerful to include a range or "ladder" reference to a part or parts of the SGML element tree by position. A very useful fragment ID for plain text would allow ranges to be quoted by line and character number

These things are all decisions made when the MIME type is defined.  Therefore,

The fragment ID spec for a new MIME type should  be part of the MIME type registration process.

Different MIME types then can have different fragment ID specifications. When HTTP for example negotiates between different content types, it is clearly useful for those types to have a consistent (hopefully identical) fragment ID syntax and semantics.

Fragment identifiers for RDF identify concepts

The semantic web has information about anything. The fragment identifier on an RDF (or N3) document identifies not a part of the document, but whatever thing, abstract or concrete, animate or innanimate, the document describes as having that identifier.

It is important, on the Semantic Web, to be clear about what is identified. An http: URI (without fragment identifier) necessarily identifies a generic document. This is because the HTTP server response about a URI can deleiver a rendition of (or location of, or apologies for) a document which is identified by the URI requested. A client which understands the http: protocol can immediately conclude that the fragementid-less URI is a generic document. This is true even if the publisher (owner of the DNS name) has decided not to run a server. Even if it just records the fact that the document is not available online, still a client knows it refers to a document. This means that identifiers for arbitrary RDF concepts should have fragment identifiers. This, in turn, means that RDF namespaces should end with "#".

Object Names as fragment identifiers

When a document language (MIME type) has some form of intra-document naming for objects then it is intuitive is these names can be directly used as fragment identifiers. This is true for XML, that the XML ID which is used to identify elements can be directly used as a fragment identifier.

Fragment IDs and Content negotiation - known bug

If content negotiation occurs across types which do NOT share a fragment ID specification, then rigidly there has been an error. In practice, HTML was the only type (in 1997) which allowed fragment IDs anyway, and other types ignore it. Also, as falling back from a pointer to a specific view to a pointer to the whole document has been considered effective fallback procedure, so no harm was done. Now (2001) it becomes more of a problem. there have been proposasl to add the requested fragment idntifier to the HTTP request to fix this.)

In the future, metadata returned or warnings returned should indicate to the client that this could be a problem. Also, in new access protocols, the fragment ID requested could be shipped to the server as a hint, which would allow the server and client to negotiate and if successful arrange for the fragment ID to be converted to a suitable equivalent value for an alternative MIME type.

User awareness of the form of a reference

Clearly when a fragment ID is generated and associated with a URI which is generic in any way (language, version, etc as well as content-type), then there is a possible failure of the fragment-id refers to something which is not defined in any specific instance.  It would be appropriate for a client, when generating a link (or bookmark, etc) to provide the user with a choice of

• A bookmark to the whole living document, or
• A bookmark to a specific part of a "dead" version;
• Intermediate combinations.
  

As both these options are meaningful and useful, they will have to surface at the user interface level.

Generic Resources

A URI represents a resource

A "resource" is a conceptual entity (a little like a Platonic ideal). When represented electronically, a resource may be of the kind which corresponds to only one posisble bit stream representation. An example is the text version of an Internet RFC. That never changes. It will always ha the same checksum.

On the other hand, a resource may be generic in that as a concept it is well specified but not so specifically specified that it can only be represented by a single bit stream. In this case, other URIs may exist which identify a resource more specifically. These other URIs identify resources too, and there is a relationship of genericity between the generic and the relatively specific resource.

As an example, successively specific resources might be

1. The Bible
2. The Bible, King James Version
3. The Bible, KJV, in English
4. A particular ASCII rendering of the KJV Bible in English

Each resource may have a URI. The authority which allocates the URI is the authority which determines wo what it refers: Therefore, that authority determines to what extent that resource is generic or specific.

This model is more of an observation of a requirement than an implementation decision. Multilevel gnericity clarly exists in all our current life with books and electronic documents. Adoption of this model simply follows from the rule that Web design should not arbitrarily seek to constrain life in general for its own purposes.

Dimensions of genericity

When we discuss electronic resources, an interesting fact is that a small number of dimensions of genericity emerge.

The fact that there are such a small number of dimensions currently apparent sugests that Web software should handle them individually in its interface with the user, even though the architecure should handle them as a single concpet.

Derivation

When a document is translated, one of the language-specific resources may have been the original source. However, this need not always be the case. Specific resources may have been derived from unrelated sources, or multiple sources. Therefore, though it is interesting to be able to describe the "derived-from" relationship, this is not part of the genericity relationship. It is not discused further here.

Genericity Metadata

When making statements about resources, genericity leads two types of statement. The examples use imaginary HTML elements or HTTP headers as illustrations of the meaning.

Dimensions

A statement about the genericity of an object is important both for the user, and also for example for a cache manager. This statment takes the form of a list of dimensions in which the resource for a given URI is generic.

One proposal was the vary field in the URI: header in HTTP:

URI: http//foo.com/bar/baz vary=time,language This is a statement about the relationship between the URI and the resource. (See also Quality of service of names)

Relationships

The other statement which can be made is about a genericity relationship between two resources. Typed links provide this kind of statement. One proposal was

 
         <link rel="language-specific" href="baz.fr">
 
 

which means "This resource is a language specific version of this resource identified by baz.fr" This needs to be combined in with information about the particualar language.

 
         <resource uri="baz.fr" vary="type, time">
                 <meta htp-equiv="content-language" value="Fr">
         </resource>
 

So much for the architectural ideas. In practice one would use a shorthand form for all this information such as

         <specific language="fr" uri="baz.fr">
 or
         <specific language="fr" ct="text/html" uri="baz.fr.html">

Using RDF to model this

There is now an RDF ontology for these concepts, http://www.w3.org/2006/gen/ont. The ontology does not describe the target-medium dimension. (Please use that instead of the old one desribed here in 2000-09.)

Old ontology RDF to model this

Now that the RDF metadata architecure is developed, we can model genericity using a set of properties to represent these relationships. The natural way to do this is to define classes for the one-parameter flags such as time-invariant, language-invariant, etc and properties such as isLangaugeSpecificVersionOf.

u:Fixed is a subclass of each of the other three. P3P policies are supposed to be in u:Fixed.

There is no assurance when one of these properties is used that either subject or object is not itself invariant. In other words, if one states of two identical TimeInvariant resources that one is a version of the other, that is consistent. The promise that neither will change can be made later as a consistent with an earlier promise that one will not change.

Read whole article...

Read whole article...

HTML and XML

The goal of this document is to investigate the possibility, over time, of healing the rift between the HTML5 and XML technologies, to achieve interoperability between software and markup which are currently on two sides of the fork.

The method is is to try to understand the motivations of the various positions, and address those at source, and not to use them to decide that a particular fork is "right".

The content of this essay is accumulated from many sources. It was given in large part as a talk to the May 2008 W3C Advisory Committee meeting, posing a series of questions about future directions for HTML. Discussion of this topic is directed to the W3C TAG list, www-tag@w3.org (archive) .

Introduction

The development of Web technology advances at different speeds on different fronts and different times. Occasionally it seems that some strategic thinking is necessary in order to ensure that the system as a whole will continue to work well and evolve smoothly. This is one of those times.

The fork

The purpose of this essay is not to detail the history, but let me start by summarizing quickly to set the context. HTML is the most widely deployed document format by a long way in the history of computing. XML, also, is very successful, being a framework for many formats public, private, in many different applications. As a simplification of the original SGML, on which HTML was based, XML allows code to be lighter and faster than SGML systems, and makes it easier for developers. We have seen in recent years a hiatus in the development of HTML, followed by a more recent surge along two branches. One branch of HTML, XHTML, which switched from using SGML to using XML, provided various new features, used the XML namespaces extensibility system, but was not widely deployed in the dominant Web browser, Internet Explorer. Another branch, HTML5, has been specified with the explicit goal of describing exactly the rather contorted behavior existing browsers implement to handle the legacy of Web pages found in practice on the Web, as well as introducing a different set of new features (video tags, etc). While it provides for an optional XML serialization, HTML5 does not in general use XML and specifically does not use XML namespaces. Below we unravel the separate criticisms of XML and XML namespaces.

The existence of the fork is a serious problem, both because a fork in standards is fundamentally costly for the whole community going forward, and because of the technological problems which are highlighted in the issues which each branch has with the other branch.

Arguments for cleaning up

Now, there may be extreme versions of the HTML5-fork style which maintain that everything is fine, and that the mess is just life; we will have to live with liberal parsers forever, and that is the only realistic approach. However, not only is the code stack horrible to maintain, but pages that are not well formed are hard to maintain, process, and reuse.

Also, there is a whole world of XML-based software in the enterprise, some of it SOAP-based services, some of it more document-oriented, whose developers could not imagine for a moment deviating from the XML path by allowing this sort of liberalness, as systems would just stop.

Can we assume that the HTML Web and the XML enterprise systems will be non-interoperable worlds? Possibly, but with a constant cost, whenever attempts are made to move data from one to another, to embed some HTML product description into an order, for example.. The boundary will never be clear as in fact there is an overlap. Some suggest making a version of SVG which is in HTML5 (liberal) format, while others use XML engines to process SVG. People embed HTML in RSS and Atom feeds and RDF feeds (using RDF's XMLLiteral datatype) and RDF parsers don't have embedded HTML5 parsers, so it has to be well-formed. And so on.

To continue to promote messy code on the Web is to create problems and pain later on. To promote clean XML is a current pain for real users which they will not put up with. How can we escape from this? To understand possible paths forward let us look at how the language is typically extended, on each fork.

Centralized HTML extensibility

The HTML community has not embraced URI-based extensibility. In fact, decentralized extensibility is not a general goal to many. This is not surprising. HTML is the most widely deployed data format in history by a long way. Every Web browser is expected to be able to handle it. Its evolution is a form of ongoing negotiation between users, Web developers, and browser developers (and their management). The HTML language itself has a unique place among other languages. The model of a large number of small overlapping communities, which was the target of the RDF design, does not apply to the HTML language.

It is not surprising that requiring each HTML document to start with a namespace declaration irks those for whom the whole world is HTML. When everyone is deemed to know the HTML spec, why have it vectored to by the XHTML namespace and the namespaces specification?

Decentralized extensibility allows new modules to be added to a language by third parties, but why bother when the modules which are generally proposed for addition to HTML, such as SVG, MathML and XForms, can be counted on the finger of one hand? In this case the HTML design authority can simply add new modules themselves. If extensions are needed, then they can just be added to the specification. The list of modules can be made available to everyone, as all systems are expected to be programmed with an inbuilt knowledge of the HTML spec.

While browser plug-ins can be dynamically downloaded from the net, in general HTML extensibility from one level to the next has not been done in that way at all. Using the foundational rule that browsers have from the beginning ignored tags they did not understand, new HTML tags have been added in a calculated way so as to hopefully maximize the benefit and minimize the damage to the community as a whole. Historically, the HTML working group did not make any commitment that the meaning of tags would not change over time, only that change would be made as responsibly as possible.

Decentralized extensibility

Let us investigate the philosophy, now, of the XML branch.

In a world in which there are very many XML-based technologies, and many many groups needing to create new ones and extend old ones, a major motivating requirement has been decentralized extensibility. This is the requirement for a group to be able to define the terms involved in the new technology without having to get an audience with and agreement with central committee. (Examples of centralized extensibility include for example Dewey decimal system, the Library of Congress cataloging system, and the international phone number space).

URI-based extensibility

In the Web environment, decentralized extensibility can be done using HTTP URIs. Basically this means that any group which can lay claim to some (normally HTTP) URI space can pick a URI for a new feature, without having to go through any centralized clearing house (other than the domain name system). It also means that the namespace URI can be used to give pointers to developers, or, with very persistent caching, machines, willing to learn about the new features.

Historically, namespaces were actually a requirement on XML Namespaces imposed by RDF, which was developed in parallel with XML RDF is aimed at a multitude of communities all independently agreeing on different though connected set of terms, and then being able to merge their datasets which use these terms. URI-based extensibility has been very successful in the world of RDF itself, as many ontologies have been developed without central coordination by the RDF working group, which indeed closed long ago. One might argue that arbitrary non-RDF XML applications cannot use URI-based extensibility in the same way, as they do not have the very powerful "ignore triples you don't understand" model of RDF, but a counterexample would be the use of independent namespace-qualified tag names in SOAP messages, headers and content. Another example would be the EXSLT group which uses use namespaces to extend XSLT.

Follow-your-nose principle

The use of HTTP URIs for extensibility is not just a question of allocating names unambiguously. The fact that HTTP URIs have ownership means that there is a responsible authority who can be traced and called upon to explain what a term is supposed to be for and how it relates to other terms.

In fact, as we use HTTP URIs, one can in real time look up that information. Although, for the sake of the servers, the looking up of a namespace document should be viewed as an installation process with a permanent cache, a machine can usefully pick up information at run-time which will allow a system to usefully process a vocabulary which it has not before encountered. This again is much more developed in the RDF world, where ontologies can contain enough information for a new user interface to be created on the fly.

The follow-your-nose principle, then , allows a form of bootstrapping. Like any bootstrap, though, it needs a base to start from. In this case, there is a core set of specifications which a client has to understand in order to do the bootstrapping. Examples of these core specs are Ethernet, TCP and IP, DNS, HTTP,, the Internet Content Type (also known as MIME type) registry.

By one model, a content type of text/html in a HTTP response indicates an HTML document. A content type of application/xhtml+xml indicates an XHTML document.

By another model, a content type of application/xml indicates an XML document, and if, within such a document, namespaces are used for the document element, then the XHTML namespace URI (http://www.w3.org/1999/xhtml) within it indicates an XHTML document.

Recent controversies

Recently, controversies have arisen as various groups have attempted to create new feature sets suitable for adding to HTML and similar languages. One of these is ARIA, which allows a Web page to be annotated to explain the user interface function of various elements, and another is RDFa, which allows a Web page to be annotated to explain the meaning of various elements and add more data. Each of these technologies, like many other technologies one can imagine, works by adding new attributes (and sometimes elements) to the markup.

In ARIA, about 30 new attributes are added. In the XML fork, in one design, these were added using a aria namespace, as, say, aria:foo, while in the HTML5 fork, they were added as aria-foo in the HTML namespace. The arguments about these choices were fairly long and complex, and involved for example discussions of what exactly legacy browsers would do with the DOM in each case. The users of the spec are not just document writers, but also those who write scripts to access and interpret the attributes. In any event, there was no way one could write the same thing in both language both at the markup and the script level, it seemed.

In RDFa (derived from "RDF in attributes"), the requirement was to add new attributes to allow semantics to be given for embedded HTML data. The GRDDL specification, an existing recommendation for pointing to a transform script which extracts RDF from a document, is a possible point of leverage in the follow-your-nose story, if one takes GRDDL as being, for semantic Web clients, as being part of the bootstrap core functionality.

In the XML fork, extensibility is achieved using namespaces, but in the non-XML fork, there are a number of less obvious options, which include the addition of all new attributes to the the HTML world, as though they were in the HTML5 spec. In this case, the social question is: can a group just announce that it is adding attributes to the HTML namespace*, or does it have to get it put there or at least agreed with the HTML design authority? In the normal world of standards, the latter is the rule, as each specification needs, it is felt, a coordinating body. In the HTML world, though, introduction of new tags by vendors, and new attribute values (such as rel="nofollow") is often done without such coordination; the 'marketplace' decides which tags live and which don't, and the low probability of collision replaces the use of clearing houses for new names and values. [fn1]

In practice, then, ARIA and RDFa have proposed to add new attributes (and/or elements) to HTML, deeming them to be added by dint of the existence new specifications, seeing whether they get adopted by a community of readers and writers once specified, and seeing whether they appeal to the those involved in the mainstream HTML language evolution to be worth either inclusion or reference.

So, can we just use a different model for HTML, because of its special place among languages?

While these are two recent examples, one soon discovers many examples of the development or integration of new technologies in this area:

The SVG community has made a very modular specification intended to be mixed with other markup languages, originally using namespaces.

The Mobile HTML specs have used XHTML very cleanly, and XHTML has been integrated with SVG in some cases, following the XML fork. SOAP systems enclose all kinds of XML in their payload, and can include XHTML within that where textual data is present in a remote service invocation or response.

Meanwhile, by contrast, suggestions have surfaced that SVG should be integrated into HTML5 simply by pouring the SVG tags into the HTML specification, using no explicit extensibility controls at all.

So it is impossible to draw a line around HTML as a special case isolating it from the mass of different communities developing their individual applications. So what can be done?

Scale free space

The Web is, as I have mentioned before, composed of many different communities of different sizes, and often is seen to have a scale-free properties. That is, for example, that there is no 'typical' number of inbound links to a page, but the distribution follows a power law. This is partly a measured phenomenon of the Web; it is also a phenomenon which occurs in many other systems, and also I have an unproved hunch that it represents a form of optimal arrangement for society to function effectively. It may be the optimum tradeoff between the ungainliness but great interoperability of a central language and the agility of small communities using of a Babel tower of different languages.

It is a characteristic feature of such scale-free systems that they have one leading player, closely followed in the popularity ranks by other players in decreasing popularity.

In the case of vocabularies on the Web, we have the HTML as the largest scale, in which tags are just tags and everyone is supposed to know them. One could argue that SVG actually belongs at this level and should be and will be as widely deployed as HTML.

At the next level we have languages which are not HTML but still address the needs of very large communities. SVG, MathML and ARIA are examples.

There are many medium-sized communities. The FaceBook Markup Language (FBML) is an example of a vocabulary proposed by one website, though a significant site. Atom feeds for various things can be considered at this level. Also, enterprise systems include many many XML namespaces which are developed, for example, in SOAP-based applications.

Continuing on (roughly) down the scale we get to vocabularies for protein scientists and history museums, for scout troops and bird fanciers; we get vocabularies invented for today's experiment in a lab, for the import of a particular spreadsheet and so on.

It is reasonable for us to not just sit back and admire the scale-free nature of the space, but to actively engineer for it. What does this mean in this case? It means that we should engineer the system with an understanding that HTML is a dominant language (at the moment) used by a very large community of individuals, but with an understanding that there are many other communities, many other languages and specifications, and that these often have to be able to connect with the HTML architecture.

I would like to investigate the possibility of us deliberately designing ourselves a system which is optimal, in that it addresses the needs of all parties, and brings the two branches of the fork into the same space, so that that there is a continuum of extensibility. We start by looking at the issues and problems with that arise when attempting to use XML and namespaces as the basis for the HTML5 fork.

XML Issues

So what are some of the issues with XML which drive the HTML5 fork away from becoming closer to the XML fork?

At the top are the ones which one could imagine being cured by a redesign of XML. To the bottom are the things which I would resist changing in HTML. In the middle are areas where one could imagine some compromise.

One fundamental difference of philosophy between the forks has been the attitude to deviations from the specification. In the past, people making Web pages have made many deviations from the specifications, so long as they worked. The result is a a legacy of Web pages which have all sorts of errors. It has been essential in the market for browsers that they work with these pages. The approach taken in HTML5 has been to document the behavior of these browsers, so that everyone knows what it is. The goals is that all old pages still work, but there can now be a well-defined algorithm and a test suite, instead of a heap of connected kludges implemented separately at great cost by each browser maker. This world is, then, very liberal, in what the Web page writers are allowed to do, and in what the client software has to accept.

The initial approach taken in the XHTML fork was very different: it was completely conservative. Recognizing that the situation which had arisen with legacy HTML was a big mess, XHTML started anew. A new content type was allocated for XHTML. The XML specification required that any processor deliver no results if the input was not well-formed XML. The idea was for XHTML to start a new branch of clean content which would eventually outgrow the old, and which would be a platform for much cleaner growth, with namespace-based extensibility, and addition of SVG, MathML, XForms, and enterprise-specific extensions in a well-defined way. Organizations and individuals who have adopted XHTML are often vocal in their praises for the benefits which they experience, but this has evidently not lead to any substantial inroads into the dominance of HTML in the general public web.

Robustness Principle

The Internet specifications, since RFC793, have been developed with guidance from a principle that one should be conservative in what one generates, but liberal in what one accepts. This is often a useful maxim, when writing a program to send or receive messages, and when there is a possible area of the spec open to interpretation. So one would send lines of limited length, but accept lines of any length; send always the same case as in the examples, but accept either upper or lower case, and so on.

This maxim works when two programs are communicating with short-lived messages, and when there is feedback between engineers when a system doesn't work. It has not worked so well on the Web, because the Web page designers in fact paid no heed to being conservative. They were not in general engineers who had read the spec at all, but random people copying each other's Web pages, and seeing what worked when they modified them. Further, the Web pages have a long, hopefully very long, lifetime. Once a Web page is out there with badly nested tags, it is out there for good. So on the web, there are some page creators who are no longer present, and others who are around and are open to feedback, new languages, and constructive feedback. Should the robustness principle be used or, if not , what?

Incentives

To look at a system which includes people, one must study the incentives for those people. Suppose there is, on the one hand (and on the X axis) a certain effort which a Web page author puts into the writing of a Web page, to eliminate various levels of error, and on the other hand (and on the Y axis) a reward given, in part, in terms of the quality of the rendered Web page on the range of clients perceived to be of interest.

In the case, shown above, of the conservative, XML fork browser, the page must be completely correct or nothing is rendered. The writer who has an almost perfect page is motivated to fix it, but the writer who has a page with several errors is not, as there will be no noticeable reward for incremental improvement. It is not very surprising that the majority of Web users whose pages would have started off near the left of the graph did not make it to the right when serving their code as XHTML.

Some errors we may consider hopeless even in HTML, in that no useful recovery seems possible for them. In the case of the liberal browser (above), the reward for a hopeless page is zero, but for a page with any other level of errors, it in fact is rendered completely by the browsers. Therefore, a writer whose page is hopeless is motivated to clean it up a little bit. But the writers of pages which have other levels of error are not motivated to clean them up at all.

So while the liberal and conservative forks have very different philosophies, they share one thing: They do not motivate the writer of a Web page to progressively improve their offering.

Bringing the fork together

The solution, as I see it, is to look at the motivating slope and fix it. When the user is provided with incremental rewards,

then he or she will move, hopefully, up the slope.

Motivating slope

What does this mean?

It means distinguishing more than the two possible outcomes of success and failure. We need to make a slope, so we need different levels.

It means recognizing all the errors as errors, but also allotting them an importance level, so that users can concentrate on fixing the more important ones, or perhaps the ones which give the best improvement per effort ratio.

There has been push-back against the idea of showing error indicators on Web pages, because no one wanted to be the browser to give the sub-optimal user experience. This can change in several ways. It can change because of user attitude changes. Al Gore points out repeatedly that we need to clean up the planet. People understand when we have to do some clean up. A browser which does not have these features would be seen as irresponsible in this context.

So step one is to have a tool bar which slides down when a page has errors, giving a rating to the page out of 100, and allowing drill-down by interested users. It is true that most users are not interested and are not able to do something about a random site they visit. However, they might still be interested in the fact that the site is not clean. People who buy a business may be interested in knowing whether that business pollutes the planet. Similarly, people may be interested in knowing whether the HTML that they publish or the sites that they visit are polluting the Web.

Another possibility is to allow users to specify which Web sites they are connected with. Anyone involved in the production of a Web site (up to the CEO and board for the company!) should be able to put that site into the list of sites for which they want more detailed feedback.

Changing the browser

We can also be smarter. We can make it so much easier for people to do the right thing.

The classic way the Web spreads is by the "View Source effect" . You like someone's Web page , you do a View Source operation in the browser, and then you copy it and paste it into your own Web page. This is the way Web technology has spread, and also the way of course all those problems have spread. Suppose, whenever I look at the source of a page, I see a cleaned up version? Suppose it is impossible (or very difficult) to actually see the original source without it being heavily marked with the places where it has syntactic errors. Suppose if I copy it in a clipboard, then I get the cleaned up version? Suppose this applies to "Save As" too? The code to clean up a Web page is not that big by today's standards. There are many implementations, Dave Raggett's tidy being a well-established one, also now as in Marc Gueury's HTML Validator Firefox Extension.

(One way to do it of course is to simply re-serialize the DOM tree of the page as loaded. This of course loses the formatting, which in general is a disadvantage, particularly when one needs to compare versisn of source files, or use source code control systems whcih do so).

It wouldn't have to be perfect. It would have to move a page substantially along the curve toward the clean end of the spectrum.

There are some things which browser manufactures could do right now, which could in fact change the ecosystem of developers and pages so that in a year or two a significant number of new pages were being produced cleanly, and in a few more years as the new content starts to dominant, the majority of the pages you see on the Web would be clean.

We are not talking here about a switch to application/text+xhtml, but continuing to use the MIME type text/html and progressively improving the content we produce that so that it becomes cleaner.

Would that be a good idea, and what exactly would it mean?

Well, in fact if everything was XML some people might regard it as actually less useful than the current HTML, when it comes to quotes around attributes. So this forces us to look at whether XML could actually change itself, to meet HTML between their current positions. So I would suggest that some of the things we would have put on the slope, some of the cleanliness goals, we simply remove and declare them non-goals. But to do that, we have to change XML.

Changing validators

It turns out a that the opinion of the W3C validator has a large amount of clout in the community. Specifications such as microformats and ARIA have been affected very much by what can be done without breaking validation. Now the validator to date has been a DTD-based validator, so it checks that the document conforms to a given grammar. It requires, to be happy with the page, a DTD declaration which specifies what grammar the author of the page thought the page was written with.

DTD validation will not allow the normal forms of XML extension, the addition of new elements and attributes. This is very ironic in a way. The "X" in "XML" is for "Extensible". The whole point is that an application written in XML can be extended by adding new element and attribute types. With namespaces, these elements and attributes become grounded in the Web, and URI space provides a way of avoiding any collision.

In this vision of a way forward, validators, or perhaps one should say page checkers, as validate is a word claimed by XML for DTD-validation, should give a grade to a page, judging it on several counts, at various levels. At the error level:

• Content-Type wrong
• Character encoding (if marked UTF-8 is it really UTF-8?)
• Well-formedness: Bad nesting, missing end tag
• HTML elements misplaced according to some kind of grammar

At a warning level:

• Extension tags used with no namespace
• Extension tags used, in a namespace without a namespace document

At an informative level:

• Extension tags used with a namespace and namespace document
• Extension tags defined in other W3C recommendations
• Quotes missing from attribute values which do not contain spaces

In fact, it may be that the browser, now a computing platform of some power, is in fact the best development platform for a page check in the future. It is possible that the same code in fact could be deployed in a third party server checker harness as in a client-side checker.

Changing XML syntax

The arguments against changing XML are very strong. Its single great value is its single common specification, its stability. It isn't perfect but it is common across so many different applications. Attempts at create an XML1.1 failed, just trying to introduce a few new Unicode characters.

The arguments for changing it are that the alternative could be worse: It could be that the HTML5-style syntax with errors of all kinds being completely ignored propagates into first SVG then RSS then RDF and then SOAP. The entire stack has to be built so as to be able to do HTML5 error ignoring, with special knowledge that comes with that of various HTML tags. Even if you aren't using HTML itself, you just have to use that parser.

What would be changed? It would be recommended that parsers recover from errors where they can, and indicate all errors above a certain level of seriousness to the user.

Now everyone I have spoken to about this has their own list of things they would like to change in XML, if we were to do it, so deciding what goes in would be a interesting communal decision. Here is a list of some things which have come up. Some are better ideas than others, in my humble opinion.

• Allow attribute quotes to be omitted for simple values.
• Allow namespace to be implicit, given Content Type
• Short-hand for switching from one namespace to another? (grounded in Namespace document)
• Short form of close tag </> ?
• Remove DTDs.
• URIs for grammar, cross-schema links for mixed NS
• Remove PIs (Have a xml:pi element if you like?)
• Multiple root elements or mixed content as document?

(See Tim Bray 2002,Norm Walsh on XML2.0 in 2004, 2008),...

Lets go through these in order to clarify what we are talking about.

Optional Quoting of Attributes

The quoting of attribute values I have already mentioned. The quotes in SGML were not necessary. When SGML was simplified to XML, the quotes were made mandatory. this simplified the parser, but it complicated life for writers, and required more keystrokes, disk space and bandwidth. It also made the source more difficult to read by increasing clutter. It also made the source more difficult to read

Implied namespace

The implied namespace idea comes from a consideration of the follow-your-nose argument above. If an HTML document is delivered with a Content-Type which labels it as HTML, then why on earth does this information have, in XHTML, to be repeated in the document as the root namespace element? It is a waste of space and an imposition on the user. However, whether the page has an explicit namespace or not, I would like to be able to parse it and look for elements in the DOM using the XHTML namespace. So I would like all HTML elements to be deemed to be in the XHTML namespace. This is actually I think a sensible change to the architecture, that:

1. With XML-based content, the MIME type registry contains an implied namespace. for text/html, this is the XHTML namespace
2. The XML parser interface is extended to include an extra parameter, the implicit namespace

(Note that while this is a default for the namespace, as the term default namespace already means the namespace for elements with no prefix, we can't use it for this concept of the namespace for the default namespace when there is no namespace declaration.)

This would make SVG documents smaller as well, and who knows what else. It could be useful for cutting down the transmission time for small XHTML documents to mobile devices, and so on.

What about mixed documents? Well, the HTML mime type could be registered to that the implicit default namespace is XHTML, but also there is an implicit s: namespace for SVG and m: for math, and so on. A machine-readable list could be made centrally available, changed occasionally, an downloaded at install time (not run time, to save the servers!) in XML-2 parsers.

Switching namespaces

The fact that documents have a well defined meaning as grounded in the Web traces back to the terms being defined using URIs. This does not, however, mean that URIs have to be embedded in their full glory at every step. Namespaces already serve as an abbreviation system. Now we could add a sort or chaining within documents. For example, one could define an <svg> element in the HTML namespace which would have the implicit effect of switching namespaces to the SVG namespace.

I am not sure that this is a good idea, as it makes the amount of off-line information needed more complex, as one would have to have a way of specifying this in a schema language of some sort, and it would be impossible to parse the document correctly without that schema document. But it could be valuable avenue to explore if there continue push-back against namespaces.

Remove DTDs

The DTD syntax within XML is a historical artifact. It was part of SGML, used for defining grammars of SGML applications, but not itself using SGML syntax. The DTD language was kept in XML as at the time there was nothing to replace it. Since then, DTDs have been joined by XML Schema, Relax-NG, and other languages for specifying constraints for applications of XML. Meanwhile, DTDs have fallen behind in that they do not naturally accommodate namespaces. A large amount of infrastructure has been constructed around them in the XHTML fork's HTML Modularization spec.

The main reason for keeping DTDs in XML systems has been that they are needed for defining entities, and specifically character entities. The solution to this I would suggest is to define a namespace of tools to do this in XML. One could even take part of the xml: namespace.

This would also mean that one would lose the feature of default values for attributes and fixed values for attributes. These are a strange feature of the language in many ways. They make this unfortunate difference between a raw infoset and a post-validation infoset. They allow the DTD designer to say "even of you didn't put this attribute in, you still meant it". Of course the semantics of the application language can always be defined to have defaults, even when they are not provided by a DTD processing step.

There have of course been many discussion of this topic over the years.

Processing Instructions

Processing Instructions (PIs) are a strange corner of the XML specification which could be removed to advantage. PIs provide a form of "machine-readable comments" which sit between code (normal markup and text) and comments (which should be completely ignored in the application semantics).

Where one is tempted to use a PI, one should use a namespace to add an attribute for example to the root element. That allows one to have many levels of hint to different possible processors and interpreters about different things. After all, why have three levels when you can have n? (In fact, in RDF, I would often recommend that comments be left as rdfs:comment statements so that they are preserved in the processing and enlighten people reusing the data in a completely different contexts)

PIs are a kludge. The question of what is inside them, and what it means,

Close tag abbreviation

A commonly suggested shortcut, while we are discussing shortcuts, is to allow the closing tag </foo> to be given as </>. I understand this was a question of debate in the original XML design, and did not get in at the time. It is less self-documenting and less robust in the face of certain errors, but it can save a lot of space for enterprise applications where tag names can become very long. Also, for machine-generated code where operator error is not a problem and indenting can be done automatically, it clearly cuts down on the size of the file.

Multiple root elements, or mixed content as XML document*

A characteristic which XML does not currently have is the ability concatenate two valid XML documents and get a new big XML document. This property would have its uses. To make it possible, one could allow mixed content (A mixture of elements and text) a the outermost level. Advantages include that

• It would be possible to transmit arbitrary XML content (for example from a selection, or the answer to a question on a form) as an XML document itself.
• One could concatenate XML documents and ship them as one when the information to be transferred was the union of the information in each;
• One could XML as a format in which the default was plain text, but markup could be allowed if necessary. For example, the title of a book, often plain text but sometimes needing a character entity, or a form field or e-mail in which the default might be plain text but occasionally one wants to add HTML emphasis.

What source could look like

The markup for a page which currently in XML will typically start like

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC
 "-//W3C//DTD XHTML 1.0 Transitional//EN"
 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
  <link href="../../../People/Berners-Lee/general.css"
  rel="stylesheet"  type="text/css" />
</head>
...

in the future when served as text/html could look like simply

<html>
<head>
 <link href=../../../People/Berners-Lee/general.css
  rel=stylesheet type=text/css />
</head>
...

and be considered perfectly valid XML2. It could be parsed by general XML2 applications, which would be passed the implicit namespace which would have come from a content-type lookup table. To HTML authors, the only non-HTML thing they ould have to do is remember the /> on the end of the link tag. So there is in te end a compromise between the forks, but one in which everyone can do most of what they want. So this may be a better place to be. Is it worth trying to get there?

Costs of change

Changes to XML syntax would of course be a vary major step. It would break a level of stability in the XML specification which has been one of its major advantges. it would potentially affect very large number of parsers.

On the other hand, it would only affect the parsers. The XML data model is not changed by the surface changes to the syntax. XML1 files would be valid XML2, so srializers woudl not have to change. Languages such as XPATH, XSLT and XQuery which are defined on the data model would not change. However, just changing XML parsers would be very dramatic step for the industry. It would leave behind many programs whos development has stopped.

But then again, if the alternative s that all systms have two parsers one for HTML5-like data and one for XML, that is a huge cost too.

What about the cost of change to browers?

Browsers currently have very many ways of treateing web pages, to adapt to different forms the language out there on the web. In one sense, a merge fork track would be another variation, one chosen to be more stable in the long term. The tricks to recognize particular types of old content will presumably be necessary into the future.

Changes to the browsers to bring them toward a common DOM for HTML and XML are also going to be significant. To a certain extent, perhaps one can allow the namespaced API calls to follow the XML+Namespaces model, but the non-namespaced calls to follow the HTML model. The complications here are to great to go into here. ^dw

Conclusion

Future developers will not only use the languages we define today, they will build on them to make new more sophisticated ones. The cleaner the systems we develop are, then the easier it will be. The HTML and SVG document models, for example, are powerful user interface libraries, and exciting novel new applications are being built on top of them. The difficulties inolved in dealing with the different APIs of different forks doesn't help.

We, the Web technology community at large, have a duty to lead the technology toward cleaner engineering solutions. While we should reatain an ability to read old web pages, we should move the community of producers (both hand-coders and authoring tools) so that the newly produced web ages become progressively cleaner.

To do that, we have to understand the motivations of website developers and browser writers and server administrators. We have to understand how changes to the software and the specifications can tweak the way people behave. We can also set new community goals and a new community attitude about unclean Web pages, so long as at the same time we move the goal of cleanliness to make it less irksome.

The direction outlines here involves quite a lot of work. It means developing new parsers, page checkers and browsers which encourage cleanliness. It means cleaning up authoring tools. It involves solving many intricate technical details of how these Web pages look to a script in the DOM. But the alternative -- the current forked track --will be a lot of work too. Keeping both forks maintained with separate diverging code stacks. Writing scripts which explicitly check whether they in an HTML5 or XHTML environment every few lines. Developing increasingly complex new extension methods for HTML5 to emulate namespaces. As the future unrolls, porting new deevlopments, like the <video> tag, within HTML5 to XHTML, and new developments, like RDFa, within XHTML to HTML. Or putting up with the burden of continual re-invention of new functionality in quite incompatible ways, on both sides of the stack.

We need to set ourselves goals of merging the forks, with some give on each side. We need to switch from strictly liberal and strictly conservative attitudes to one in which progressively cleaner pages are considered progressively better. We need to adopt an attitude that we are going to clean up the Web just as we sometimes need to clean a bedroom -- or a planet.

Grouchy Robustness Principle

Be conservative what you produce. Be liberal about what you accept but complain about any deviations from the spec in a way to help and to motivate the producers to adhere to it better.

Tim Berners-Lee

Original August 2008, made public May 2019

Footnotes etc

This is a deliverable of TAG issue 145.

This is $Id: HTML-XML.html,v 1.5 2019/05/20 21:36:35 timbl Exp $

1. (There is a parallel with adding them all to the XHTML namespace, but this is unfortunately not a precise one, because attributes without explicit namespaces are deemed in the NS spec to be in no namespace., rather than to be in the namespace of the element. The fact that the XHTML <a> element has an @href attribute, for example, does not mean that there is an attribute xhtml:href which one could consider mixing into other languages. Some, including me, regard this as a bug in the namespace specification.)

2. (Footnote: At the schema level there are issues too. There is not space to go into those here, but to oversimplify, DTDs are broken by design as they actually don't use XML syntax; XML schema got complicated; RelaxNG is a competing standard, but still needs NVDL to enable mixed namespaces. There is no simple way to say the fundamental statements for connecting two languages such as "An SVG circle can go anywhere an HTML IMG can go".)

3. (Thanks to Norm Walsh for the multiple root element suggestion)

DW: Perhaps the biggers single obstacle is the document.write() method which binds code and markup much too intimitely to all them to eveolve separately. It is too close to self-modifying code. Ironically, it is often used to use a compact declarative form (document.write("<p><a href="../"><b>here</b>?</a></p>")as an alternative to sequence of method calls to biuld up the same thing. This is easier to write, and easier to read. If it were compiled into a data object (see E4X) this would be clean coding. As it is, the intricaied of when document.write() inserts what into what stream end up defining huge amnounts of how code is written, and allow one to do all kkinds of non-obvious things.

Background Note

This question has been addressed only vaguely in the specifications. However, the lack of very concise logical definition of such things had not been a problem, until the formal systems started to use them. There were no formal systems addressing this sort of issue (as far as I know, except for Dan Connolly's Larch work [@@]), until the Semantic Web introduced languages such as RDF which have well-defined logical properties and are used to describe (among other things) web operations.

The efforts of the Technical Architecture Group to create an architecture document with common terms highlighted this problem. (It demonstrates the ambiguity of natural language that no significant problem had been noticed over the past decade, even though the original author or HTTP , and later co-author of HTTP 1.1 who also did his PhD thesis on an analysis of the web, and both of whom have worked with Web protocols ever since, had had conflicting ideas of what the various terms actually mean.)

This document explains why the author find it difficult to work in the alternative proposed philosophies. If it misrepresents those others' arguments, then it fails, for which I apologize in advance and will endeavor to correct.

1. Web Concepts as here proposed

The WWW is a space of information objects. The URI was originally called a UDI, and originally all URIs identified information objects. Now, URI schemes exist which identify more or less anything (e.g. UUIDs) or electronic mailboxes (mailto:) but is we look purely at HTTP URIs, they define a web of information objects. Information objects -- perhaps in Cyc terms ConceptualWorks -- are normally things which

• Carry some sort of message, and
• Can be represented, to a greater or lesser authenticity, in bits

I want to make it clear that such things are generic (See Generic Resources) -- while they are documents, they generally are abstractions which may have many different bit representations, as a function of, for example:

• Time -- the contents can vary with revision --
• Content-type in which the bits are encoded
• Natural language in which a human-readable document is written
• Machine language in which a machine-processable document is written
• and a few more

but the philosophy is that an HTTP URI may identify something with a vagueness as to the dimensions above, but it still must be used to refer to a unique conceptual object whose various representations have a very large a mount in common. Formally, it is the publisher which defines the what an HTTP URI identifies, and so one should look to the publisher for a commitment as to the exact nature of the identity along these axes.

I'm going to refer to this as a document, because it needs a term and that is the best I have to date, but the reader should be sure to realize that this does not mean a conventional office document, it can be for example

• A poem
• An order for ball bearings
• A painting
• A Movie
• A review of a movie
• A sound clip
• A record of the temperature of the furnace
• An array a million integers, all zero

and so on, as limited only by our imagination.

The Web works because, given an HTTP URI, one can in a large number of cases, get a representation of the document. For a human readable document, the person is presented with the information by virtue of some gadget which is given the bits of a representation. In the case of a hypertext document, a reference to another document is encoded such that, upon user request, the referenced document can in turn be automatically presented. In the case of a machine-readable document, identifiers of concepts, being HTTP URIs, will often allow definitive reference information about those concepts to be pulled in to guide further actions.

The web, then, is made of documents as the internet is made of cables and routers. The documents can be about anything, so when we move to talk about the contents of documents we break away from talking about information space and the whole universe of human -- and machine -- discourse is open to us. Web pages can compare a renaissance choral works with jazz pop hits, and discuss whether pigs have wings. Machine-processable documents can encode information about shoes, and ships, and sealing-wax. Until recently, the Internet protocol standards out of which the Web is built had little to say about such things. They were concerned only with the human-readable side, so it was people, reading natural language (not internet specs) who formed and communicated the concepts at this level. Nowadays, however, semantic web languages allow information to be expressed not only about URIs, TCP ports and documents, but also about arbitrary concepts - the shoes, and ships and sealing wax, and whether pigs have wings. Simple semantic web application allow one to order shoes and travel on ships, and determine that, given the data, pigs do not have wings.

For these purposes it is of course quite essential to distinguish between something described by a document and the document itself. Now that we -- for the first time -- have not only internet protocols which can talk about document but also those which talk about real world things, we must either distinguish or be hopelessly fuzzy.

And is this bad, is it an inhibition to have to work our way though documents before we can talk about whatever we desire? I would argue not, because it is very important not to lose track of the reasons for our taking and processing any piece of information. The process of publishing and reading is a real social process between social entities, not mechanical agents. To be socially responsible, to be able to handle trust, and so on, we must be aware of these operations. The difference between a car and what some web page says about it is crucial - not only when you are buying a car.

Some have opined that the abstraction of the document is nonsense, and all that exists, when a web page describes a car, is the car and various representations of it, the HTML, PNG and GIF bit streams. This is however very weak in my opinion. The various representations have much more in common than simply the car. And the relationship to the car can be many and varied: home page, picture, catalog entry, invoice, remote control panel, weblog, and so on. The document itself is an important part of society - to dismiss its existence is to prevent us being aware of human and aspects of information without which we are impoverished. By contrast, the difference between different representations of the document (GIF or PNG image for example) is very small, and the relationship between versions of a document which changes through time a very strong one.

2. Trying out the Alternatives

The folks who disagree with the model do so for a number of different arguments. This article, therefore will have to take them one by one but the ones which come to mind are as follows:

1. Every web page (or many of therm) are in fact themselves representations of some abstract thing, and the URI really identifies that thing, not a document at all.
2. There are many levels of identification (representation as a set of bits, document, car which the web page is about) and the URI publisher, as owner of the URI, has the right to define it to mean whatever he or she likes;
3. Actually the URI has to, like in English, identify these different things ambiguously. Machines have to disambiguate using common sense and logic
4. Actually the URI has to, like in English, identify these different things ambiguously. Machines have to disambiguate using the fact that different properties will refer to different levels.
5. Actually the URI has to, like in English, identify these different things ambiguously. Machines have to disambiguate using extra information which will be provided in other ways along with the URI
6. Actually the URI has to, like in English, identify these different things ambiguously. Machines have to disambiguate them by context: A catalog card will talk about a document. A car catalog will talk about a car.
7. They may have been used to identify documents up till now, but for RDF and the Semantic Web, we should change that and start to use them as the Dublin Core and RDF Core groups have for abstract concepts.

2.1 Identify abstract things not documents

Let's take the alternatives in order. These alternatives all make sense. Each one, however, has problems I can't see any way around when we consider them as a basis as

The first was,

Every web page (or many of them) are in fact themselves representations of some abstract thing, and the URI really identifies that thing, not a document at all.

Well, that wasn't the model I had when URIs were invented and HTTP was written. However, let's see how it flies. If we stick with the principle that a URI (or URIref) must unambiguously identify the same thing in any context, then we come to the conclusion that URIs can not identify the web page. If a web page is about a car, then the URI can't be used to refer to the web page.

2.1.1 Same URI can identify a web page and a car

What, a web page can't be a car? At this point a pedantic line reasoning suggests that we should allow web pages and cars to conceptually overlap, so that something can be both. This is counterintuitive, as a web page is in common sense, not a concrete object whereas a car is. But sure, we could construct a mathematics in which we use the terms rather specially and something can be at the same time a web page and a car.

Frankly, this doesn't serve the social purpose of the semantic web, to be able to deal with common sense concepts and objects. A web page about a car and a car are in most people's minds quite distinct (as I argue further below). A philosophy in which they are identical does not allow me to distinguish between them. not only conflicts with reality as I see it, but also leaves us no way to make statements individually about the two things.

2.1.2 The URI identifies the car, not the web page

So lets fall back on the idea that the URI identifies the subject of the web page, but not the web page itself. This makes sense. We can build the semantic web on top of that easily.

The problem with this is that there are a large number of systems which already do use URIs to identify the document. This is the whole metadata world. Think of a few:

• The Dublin Core
• RSS
• The HTTP headers
• The Adobe XML system
• Access control systems

(I'm sticking with the machine-processable languages as examples because human-processable ones like HTML have a level of ambiguity traditional in human natural language but quite out of place in the WWW infrastructure -- or the Semantic Web. You can argue that people say "I work for w3.org" or "http://www.amazon.com/shrdlu?asin=314159265359" is a great book, just as they happily say "Moby Dick weighs over three thousand tons", "Moby Dick was finished over a century ago" and "I left Moby Dick on the beach" without expecting to be misunderstood. So we won't use human language as a guide when defining unambiguously the question of what a URI identifies. If we want to do that on the Semantic Web, we will say "I work for the organization whose home page is http://www.ww3.org.)

Some argue the the URI which I associate with someone's home page actually identifies that person. They argue that conventionally people use the identifier to identify the person. However, consider another page put together by friends who found a photograph of the same person. A lot of content filtering systems would collect that URI and put put into their list. Even though the photo had many representations which different devices could download using content negotiation and/or CC/PP (color or black and white and versions of different resolutions) the URI itself would be listed as containing nudity. The public are very aware of different works on the web, even though they have the same topic.

2.1.3 Indirect identification

You can argue that a web page indirectly identifies something, of course, and I am quite happy with that. If you identify an organization as that which has home page http://www.w3.org, then you are not saying that http://www.w3.org/ itself is that organization. This scenario is very very common, just as we identify people and things by their "unambiguous properties": books by ISBN, people by email address, and so forth. So long as we don't think that the person is an email address, we are fine. Some people have thought that in saying "An HTTP URI can't identify an organization" I was ruling out this indirect identification, but not so: I am very much in favor of it. The whole SQL world, after all, only identified things indirectly by a key property. This causes no contradiction. Perhaps I should say "An HTTP URI can't directly identify an organization". But by "identify" I mean "directly identify", and "identity" is a fairly direct word and concept, so I will stick with it.

Conclusion so far: the idea that a URI identifies the thing the document is about doesn't work because we can only use a URI to identify one thing and we have and already do use it to identify documents on the web.

2.1.4 The argument for HTTP URIs identifying a Conceptual Work

So what's wrong with the URI being taken to identify whatever the owner says?

Let's look at what we mean by identifies. When we say there is identity, that means that there is some form of sameness that we associate with the identifier. Now, for all the philosophical argument, we can never test the identity of an abstract thing. What we can test is a representation which has been returned by the server when given that URI. When we use aURI, and get back several possible representations of it, then what expectation do we have about those representations?

Take the test case that I see the web page which has a picture of a car, and I see in the URI in the URI bar in the browser. I email you the URI, "you see, the car is a Toyota?". You click on the link. Your browser shows the same URI as mine in the "URL bar" but you see a table of the car's weight, length, height, color, and registration number. We are confused. The web didn't work because you didn't get the same information as me. I expected you to get the same information, basically. That is how the Web works. That is the expectation behind every hypertext link - that the follower of the link should get basically the same information as the person who made the link. I say, "basically" because I would not have cared whether you saw or JPEG or a GIF. It probably wouldn't have mattered if you had seen a lower resolution or even black-and-white copy of the picture. If you are visually impaired, you may have been able to manage with a well-written description of the picture. But the the essential information is the same, not just the subject of the page.

So now we have put the four corners on the expectation we have of a URI -- that all representations have essentially the same information content. And what we mean by "essentially" allows in fact some wriggle room, and in the end it rests on a common understanding between publisher of the information and quoter of the URI. The sameness we are after is the sameness of information content. That is what is identified by the URI. That is why we say that the URI identifies that conceptual information content, irrespective of its particular representation: the conceptual work. Without that common understanding, the web does not work.

Some people have said, "If we say that URIs identify people, nothing breaks". But all the time they, day to day, rely on sameness of the information things on the web, and use URIs with that implicit assumption. As we formalize how the web works, we have to make that assumption explicit.

2.2 Author definition

So how can we break free of that line of reasoning? We can try throwing away the rule that a URI identifies only one thing.

There are many levels of identification (representation as a set of bits, document, car which the web page is about) and the URI publisher, as owner of the URI, has the right to define it to mean whatever he or she likes.

Well, this one is tempting from the point of view that the owner of an identifier should reign supreme when it comes to saying what it identifies. It is quite a logically consistent position to take. After all, isn't this the case with uuid's? And for a new scheme, this would be interesting. How can we do it though, with HTTP? the problem is an engineering one: I can't in practice use a URI until I have some definitive information from the publisher as to what it identifies.

2.2.1 Default

Why can't a URI default to identifying a web page until you know otherwise? Because the web is open and you will never know when you might lean some other information which will make the default incorrect. (You can't use such "closed world" reasoning).

2.2.2 Web operation

Why can't a URI identify a web page until you have done some well-defined operation -- such as HTTP HEAD or GET -- and checked for information in that? Well, that would certainly work logically. Suppose we we define a return code or HTTP header which means "abstract object requested". It would mean that every web application which deals with web pages as web pages would actually be working under an ambiguity, and RDF processors could be programmed to look for that special information. We can't retrofit the millions of web servers out there, I assume.

I feel that there is a great benefit to fixing this question at the spec level. Otherwise, what happens? I read a web page, I like it and I am going to annotate it as being a great one -- but first I have to find out whether the URI my browser is used, conceptually by the author of the page, to represent some abstract idea? Before I recommend the Vietnam War page, I have to be careful I am not recommending the Vietnam War.

There has been no way to do this before RDF, but then similarly no real need for it. (What, is this just a problem with RDF? No, it will happen with any webized knowledge representation system.). We really need to have communication in which two people use the same URI to mean the same thing. If there

We could fix HTTP so that it would return me some extra semantic headers explaining the whole thing. And in the case that the URI was deemed to be some abstract thing, I would not have the option of recommending the web page. Too bad: it has no URI.

The authors of document <http://www.w3.org/2000/10/rdf-tests/rdfcore/Manifest.rdf> certainly thought that they could use "http://www.w3.org/2000/10/rdf-tests/TestSchema/NegativeParserTest" to identify an abstract thing which is a type of software test. Now they have a choice as to what to make the server return for them when I ask for it. It returns 404 "doesn't match anything we have available". It can't really, because HTTP doesn't allow one to return a class, only a document. And if it were to return a document, then I wouldn't be able to refer to that document without accidentally referring to the class of negative parser tests.

So, we could change HTTP to make this work. We could make a new form of redirect, 343 Abstract Object, please see . . ., which would tell the client that the thing requested was abstract, and would suggest a document to read about it. This avenue of argument is still outstanding. We could take it. It isn't the status quo, but we could make changes in HTTP if the community felt that this was they way to go.

2.3 Logic disambiguates

Otherwise,we have to try another way of letting the URI mean sometimes one thing and sometimes another. Here is another.

Actually the URI has to, like in English, identify these different things ambiguously. Machines have to disambiguate using common sense and logic

This is possible in theory. It is a mess. It fails particularly spectacularly when a URI is used ambiguously to refer to a web page and the thing that web page is about, which happens to be another web page. Anyone can write anything about anything is a Web motto, but here it falls down. Anyone can write anything about anything except those things which might get confused with the document they are writing. It breaks the axiom that we mean the same thing by a URI - in all contexts. (And RDF has a model theory in which necessarily in any interpretation, a symbol always denotes one thing).

2.4 Different Properties

Actually the URI has to, like in English, identify these different things ambiguously. Machines have to disambiguate using the fact that different properties will refer to different levels.

One way of getting here is to start by considering that HTTP headers can be divided into those which refer to the representation (or the document) and those that refer to, say, a car or a donkey. We can look at all RDF properties and other attributes in other languages and divide them in in such a way. So, when I say "http://example.com/albert is a color photo", I am referring to the representation; when I say "http://example.com/albert used to work down the mill" I am referring to the person; when I say "http://example.com/albert was taken on a rainy day" I am revering to the original photograph, which is basically the representation of Albert.

This one has the problem when a web page refers to a web page. It can still be pursued, by having different verbs for talking about ownership of the web page and ownership of the car. This is a classic example of the 2-level syndrome (see also Dictionaries in the Library). The basic fallacy is that you can make the system general by introducing a second level - a new set of attributes, properties, or whatever, which allow you to refer to the metadata of something separately from the thing itself. These systems either turn out to be just limited 2-level systems (like XML and DTDs) or have to be extended to be recursive in some way later on such that in fact the two levels become unnecessary.

2.5 Extra info with URI

Actually the URI has to, like in English, identify these different things ambiguously. Machines have to disambiguate using extra information which will be provided in other ways along with the URI

This twist now relies on sending extra information with a URI. Effectively, the URI scheme has now failed to identify anything by itself. Those most familiar URIs as used by HTML sometimes suggested adding new attributes to the anchor tags of HTML documents to disambiguate a reference. I guess it would work if HTML anchors were the only uses of URIs. By contrast, they are used in thousands of places and way, many of which I am unaware. The architecture, however, is not that way: the architecture of the WWW is that a URI is a global unambiguous identifier. Not a URI and something else.

(The various designs such a WebDav's propfind which use HTTP methods apart from GET to retreive information suffer from this same problem. the information does not have a URI: it is not on the web.)

2.6 Different meaning in different context

Actually the URI has to, like in English, identify these different things ambiguously. Machines have to disambiguate them by context: A catalog card will talk about a document. A car catalog will talk about a car.

This works in the short term, when the two contexts are disjoint groups who do not need to communicate. It is in fact the current state: the groups of people who use HTTP URIs to talk about documents, and those who have just started to use them to talk about abstract concepts haven't collided yet. (Well, they have in my code. I need to be able to model the metadata about an HTTP URI as that about a document, and it being a class at the same time doesn't jive.)

It doesn't work in the long term because it breaks the axiom that a URI must identify one thing,

2.7 Change it for the Semantics Web

They may have been used to identify documents up till now, but for RDF and the Semantic Web, we should change that and start to use them as the Dublin Core and RDF Core groups have for abstract concepts.

I think that we would have to design a new URI scheme before we change things that much. That is tempting of course. But then -- building a semantic web out of what we have is tempting too. It was tempting to rehash TCP a little when making HTTP. It wasn't practical, and we would have lost a lot more than we would have gained. There is a lot to be said for using common technology. We've got an infrastructure of documents. We want to build an infrastructure of knowledge. Let's build it using the documents. We might find that the commonality with the web of human-readable information is a boon.

2.8 Abandon any identification of abstract things

An argument which surprised me is that yes, HTTP URIs identify documents, but in fact the frgament identifier must only be used to identify parts -- fragments -- of documents. This means that RDF cannot in fact use HTTP URI schemes at all. A completely different system would have to be put together -- either a new set of URIs, or RDF conventions in which the relationship to the part of a document in which something was described became explicit. In N3 this would like like

[ is rdf:referent of <#fmyCar> ] [ is rdf:referent of <#color> ] [ is rdf:referent of <#blue> ]

Of course, languages would quickly generate special syntax for this. Alternatively, the RDF system would built entirely on the understanding that we were referring always to that denoted by a given bit of document, not the bit of document itself. This would mean that there would be no way for the RDF system to refer to documents themselves directly.

This is actually a consistent way of working. It would be a change only for those people who use RDF to talk about documents as documents. We could change.

3. Conclusion

I didn't have this thought out a few years ago. It has only been in actually building a relatively formal system on top of the web infrastructure that I have had to clarify these concepts my own mind. I am forced to conclude that modeling the HTTP part of the web as a web of abstract documents if the only way to go which is practical and, by the philosophical underpinnings of the WWW, tenable.

I apologize again if I have misunderstood or misrepresented other's arguments in this process of this explanation of my own position.

Tim Berners-Lee

2002-07-28Z

Abstract

HTTP URIs, in the web architecture, have been used to denote documents -- "web pages" informally, or "information resources" more formally. However, with the growth of the Semantic Web, which uses URIs to denote anything at all, the urge to use and practice of using HTTP URIs for arbitrary things grew steadily. The W3C Technical Architecture group eventually decided to resolve the architectural problem that if an HTTP response code of 200 (a successful retrieval) was given, that indicated that the URI indeed was for an information resource, but with no such response, or with a different code, no such assumption could be made. This compromise resolved the issue, leaving a consistent architecture.

Introduction

HTTP URIs, in the web architecture, have been used to denote documents -- "web pages" informally, or "information resources" more formally. However, with the growth of the Semantic Web, which uses URIs to denote anything at all, the urge to use and practice of using HTTP URIs for arbitrary things grew steadily. The Dublin Core project, one of the first RDF vocabularies, and later Friend of a Friend, and various others simply used HTTP URIs to identify RDF Properties. The result was that one could no longer be sure that an HTTP URI was intended to identify the web page one got when one used the URI in a browser. In fact, there was a danger of confusion is one party used the URI for an abstract concept and another used it for the web page. The author wrote a long Design Issues note about this, What do HTTP URIs Identify?. The reader is directed to read that if more detail of the arguments is needed.

This whole issue caused, until 2005, a lot of discussion in technical circles, and much heated debate. In June 2005, the TAG resolved the issue as a function of the runtime protocol response. Basically, the argument is that if you have used a URI to get a web page, then you can use the URI to identify the Information Resource which is that web page: For example, the New York Times home page, or this page you are reading now.

Resolution

The TAG resolution effectively extends the range of things one can use HTTP URIs. However, it does not allow one to simply serve a web page at a URI which is used for something else. Of course, it is a general principle of web architecture that it is useful to serve information to those that look up a URI. In the case that the URI is not intended to be used for an information resource.

The W3C Technical Architecture group eventually decided to resolve the architectural problem that if an HTTP response code of 200 (a successful retrieval) was given, that indicated that the URI indeed was for an information resource, but with no such response, or with a different code, no such assumption could be made. This compromise resolved the issue, leaving a consistent architecture.

When XML is used to represent a directed laballed graph which is used to represent information about things, then one must be able to make statements about parts of an XML document, parts of the DLG (such as RDF nodes) and of course the objects described.

In most cases it seems obvious to the human reader. The jam jar label text does not (normally) read "jam jar label text" or "jam jar label" or "jam jar" but "jam".

Take the case of a statement about a person in an imaginary syntax

<z:person id="foo">
   <head>
      <play:author>Zoe</play:author>
   </head>
   <play:name>Albert</play:name>
   <play:mailbox resource="mailto:adoe@bar.com"/>
   <play:son-name>Bill</play:son-name>
   <play:daughter-name>Claire</play:daughter-name>
   <play:father>
      <z:name value="Joe"/>
      <z:wrote href="#foo">
      <z:friend resource="#foo"/>
   </play:father>
</z:person>

The XML element has one attribute and four child elements. The RDF node has three properties (stated here). The person Albert has two children. What so we refer to is we refer to "#foo"? Of course we refer to the element - but when we make RDF statements, we normally want to refer to the RDF node, or rather the object described by the node, in RDF terms the resource.

Of course, in a typical unix programming language we would simply add a syntax character to distinguish the forms of reference: #foo would be the node, and @#foo (or something) would be the object refered to. But in this case we are trying to do everthing with RDF, and what is left with XML, and so we would lose a few points by adding instead some totally new syntax. What we can do is to use different attribute names for the different forms of reference. The attribute names I used above are as follows:

Here I have used "href"to allow RDF to refer to the XML model. This is important, as for example it is bits of XML which one digitaly signs, not (in sigend XML) bits of RDF. Also, it is useful for RDF to be able to talk about XML elements. It brings up the question of what an RDF fragment identifier means.

RDF and XML fragment identifiers clash

This highlights (2000/02) a bug in the relationship between XML and RDF

Consider what is identified by

http://.../foo.rdf#bar

when ...foo.rdf contains among other things the following:

<rdf:description rdf:id="bar">
   <rdf:type resource="...#person">
   <y:common-name>Ora Lassila</y:common-name>
   <y:mailbox>ora.lassila@research.nokia.com</y:mailbox>
</rdf:description>

The meaning of the fragment identifier is taken from the specification assocaitedwith the MIME type.

Therfore, if this is takes as a document of type application/rdf, then the fragment identifier identifies the thing (person in this case, Ora) described RDF node. This is how refernces are used in RDF.

However, if its considered to be of type text/xml then the fragment identifier is defined bythe XML spec, and so references an element whose attrubute XML:ID. has value "bar". It happens that the rdf:id is not defined to be an xml:id but is defined to "act like one", whatever that means, by the RDF spec. So it isn't clear whether the reference to this would be to the XML subtree (consisting of the rdf:description element and its contents) or would be undefined or possibly a refernce to some other element which happened to have id="bar".

To have a different interpretation of a URI as a function of the notional type of the document belies the fact the point of using XML syntax for RDF was that RDF documents should be XML documents! Of course we embed RDF in regular XML documents. So this distinction is nonsense.

Of course, the RDF spec can simply use the XML definition indirectly and refer to the RDF ndoe described by the XML element. Howvere, this is not powerful enough for RDF. This is because RDF needs to be able to make statements about XML documents and XML elements. So for example, I might want to state that I wrote the above snipet. It would be very tempting to write that I am the author of foo.rdf#bar. But I am not the author of Ora Lassila. RDF uses and parseType to resolve this for inline data: parseType=Resource indicates that the reference is to the RDF object, and parseType=Literal indicates that it is to the XML. The thing could be resolved with an interpretion property which expresses relationship between an XML subtree and an RDF object which it describes. While it would be good to define that property, RDF syntax needs a shortcut. I would propose that "resource=" which is used to point to a resource be also used for a resource fragment id, and that a new syntax be introduced to refer to the actual RDF node. maybe "object=" which happens here to correspond to the (subject, predicate, object) sense -- as well as a "thing" sense. (The former is what is the reason for chosing it - the attribute should express the relationship, not the class of the thing refered to in general!).

Naming properties and elements

We have a similar problem in the XML-RDF relationship looking atthe identity oat the schema level.

In RDF M&S 1.0, a property name defined in a namespace is formed by directly concatenating the namepsace URI with local tag name of the XML element.

One natural way to use this is to end the namespace URI with "#" so that the local tag name becomes the fragment identifier. When the schema is written in XML, this implies that the tag name, being a simple alphanumeric, will identify something in the document by its XML ID. This is a constraint on the schema language: the XML ID of an element must be usable as a reference to the thing being defined.

When there is a 1:1 mapping netween RDF properties and XML element types, there is a choice of

1. giving them the same URI and distinguishing which is refereed to by context (as in resource= and object= above), or
2. giving the different URIs algorithimically related, like assuming that #foo-element means the element defining #foo, using a convention specified in eth schema languages, or
3. giving them totally distinct URIs which can be connected by an assertion in the schema, or an in

Given that it is interesting to use RDF to make statements about XML element types, having different names it appealing. As writing down the relationship every time the algorithmic link is un appealing.

A generic problem with XML identifiers

(I notice in passing that XML has currently a mixture of identifier paces which is a little confusing.

The element and attribute namespace is very well handled in terms of abbreviations, and is grounded in URI space, using the XML namespaces spec.

The URI space is of course the same space, but when value is typed as a URI, then it cannot use the abbreviation system of the elelemnt namespace.)

IDREF considered harmful

The local identifier space is a subset of URI space. When an attribute is defined as a URI, the simple "#" prefix gives access to the local ID space - while still allowing great pwer of expression by reference to anything else on the Web. When the "idref" form is used, this is not possiible. The idref form is a weak form IMHO and not wise for new designs which are not to be deliberately constraining.

Others have noticed this problem and there have even been suggestions which confused the URI prefix and the namespace prefix. In fact the problem can be solved [ref eric whiteboard] with an escape of some sort. One prossibility is ambushing a void URI schme name by using a colon prefix (suggested by Eric Prud'hommeaux)

href=":rdf:description"

would be a perfectly valid URI (in an XML context) which referenced the rdf:description URI using the defined rdf: namespace. I feel this is messy, as it would have to be subject to different handling than any other URI: its expansion would be done in an XML-specific way.

The other link you need is the ability, when using an element name which only occurs once, and without changing the default namespace, it would clearly be logical to be able to just write

<http://foo.com/schemas/memo6.2#priority>a</[...]>

Because what follows uses the full power of what precedes with generality, we may need to see the first in use before the paper is over. But I can't see making the second change to XML.)

What, many people ask, will happen when this huge mass of classical logic meets its first inconsistncy? Surely, once you have one staement that A and another somewhere on the web that not A, then doesn't the whole system fall apart? Surely, then you can deduce anything you want?

This fear of course is quite valid - or would be if all assertions in the whole world were regarded as bing on equal footing. Some imagine that an RDF parser will simply search all XML documents on the web for any facts, and add them to a massive set of belived assertions. This is not how realisic systems will actually work.

On the web, a fact may be asserted in an expression. That expression may be part fo a formula. The formula may ivolve negation, and may invove quotation. The whole formula is found by parsing some document . There is no a priori reason to believe any document on the web. The reason to believe a document will be found in some information (metadata) about the document. That metadata may be an endosement of the document - another RDF statement, which in turn was found another document, and so on.

[@@need picture here]

A real system may work backwards or forwards (or both). I would call working forwards a system which is given a configuartion page to work from which in turn points to other pages which in turn are used as valid data. I would call working backwards a system which, when looking for an answer to a query, looks at a gloal index to find any document at all which mentions a given term. It then searches thes documents turned up for answers to the query. Only when it has found an answer does t check back to see whether the data can be deriveded directly or indirectly from sources it has been set up to trust.

Digital sgnature (see trust) of course adds a notion of secuirty to the whole process. The first step is that a document is not endorsed without giving the checksum it had when believed. The second step is to secify more powerful rules of the form

"whatever any document says so long it is signed with key 57832498437".

In prcatice, particular authroities are trusted only for specific purposed. The semantic web must support this. You must be able to restrict the information believed along the lines of,

"whatever any document says of the form xxxx is a meber of W3C so long as it is signed wiht key 32457934759432".

for example

"whatever any document says of the form "a is an employee of IBM" so long as it is signed by with key 3213123098129".

Limiting inference

There is a choice here, and I am not sure right now which appeals to me most. One is to say precicely,

"whatever any document says of the form xxxx is a member of W3C so long as it is signed with key 32457934759432".

The other is to say,

"whatever is of form xxxx and can be inferred from information signed with key 32457934759432"

In the first case, we are making an arbitrary requirement for a statement to be phrased in a particular way. This seems unnecessarily bureaucratic, and more difficult to treat constently. Normally we like to be able to replace any set of forumlae with another set which can be deduced from it. However, in this case we have to preserve the actual form in case we need to match it against a pattern. This is very messy.

In the second case, we fall prey to the inconsistency trap. Once any pair of conflicting statements can be deduced from information signed with a given key, then anything can be deduced from information signed with the key: the key is completely broken. Of course, only that key is broken, so a trust system can remove any reason it has to trust that key. However, the attacked system may not realize what has happened before it has been convinced that the sun rises in the west.

Is there a way to limit the domain of trust in a key while allowing inmformation to be processed in a consistent way throughout the system? Yes - maybe - there are many. Each KR system which uses a limited logic does do in order (partly) to solve this problem. We just qulaify "can be inferred" be the type of inference rules which may be used. This means the generic proof engine eitehr has to work though a reified version of the rules or it has to know the sets - incorporate each proof engine. Maybe we only need one.

Expiry

Tortoise: What's the time, Achilles?

Achilles: Five past ten, my friend. [They chat for a minute]

Tortoise: What is the time, Achilles?

Achilles: Six minutes past ten, Mr. Toroise.

Tortoise: But Achilles, you just told me just a minute ago it was five minutes past ten. How can I ever believe you again?

Time-varying information is one cause of apparent contradiction. People and documents change status. How does one base inference on information which may be out of date?

One part of this is to put explicit or implcit expry dates on everything. Whenever a server sends resource to an HTTP client, it can give an expiry date. The client can track this, and ensure that all deductions from that document are cancelled when the date arrives, unless a more recent copy can be optained which says the same thing. In human language you might say "It is rainy" but on the semantic web that woudl be exported in a fully qualified way, more like "at Mon Jan 24 09:41:06 EST 2000 the measurement guage 5 at Dubin Airport read rain as having fallen in the last hour". (A fuzzy system would conclude "Dublin is wet" and a clasic logic system "at least once it rained at at least one place in Dublin"!)

I understand [Lehrmann, SW meeting in DC] (sp?) that the KIF folks developed a complete vocabulary for time-variance.

Another tchnique is to make any looseness which exists in the real system visible. Instead of saying

Any employee of any member orgainzation of W3C may register

you say formally to the registration engine

Any person who was some time in the last 2 months an employy of an organization which was som etim ein the last 2 montsh a W3C member may register.

In other words, if an organization were to drop its membership, the system doesn't have to support propagating that information instantly.

I think there will be time-aware reasoning systems, and time-unaware raesoning systems which are fed data with expiry dates and whose results are used within the intersection period of the validity periods of the incomming data. Indeed, time-aware systems may contain nested time-unaware systems, and probably vice-versa.

We need some philosophy as a basis for the architecture of digital signature and the semantic web.

The semantic web is a computer system, a distributed machine which should function so as to perform socially useful tasks. There will be various interfaces between the Semantic Web (SW) world and the social world of people, such as the physical delivery of goods, and the presentation of a document to a person for signature. However, in general with these important exceptions the Semantic Web will form a self-sufficient loop. The semantics of anything on the SW are then defined either in terms of more stuff on the SW, or in terms of the connection with these real-world connections. So for example I might initially define a check as something which when fed into the bank's black box will make it do a certain thing. Then within the SW all definitions of dollars and transfers can be defined back in terms of the check, and a self-sufficient system can be made where is necessary the recourse can be made to sending a check to a bank, but in fact we can etrade using ecurrency and einvoices and edeliverynotes and so on.

This is a similar relationship with reality that coins originally had with gold, and bills with coin. (A UK pound used to read "I promise to pay the bearer on demand the sum of one Pound signed, signed: Bank of England"). From then on a pound note became what people thought of as a pound, and the notion of what exactly the "sum of one pound" was originally defined by becomes irrelevant and the paper money is self-sufficient. So we are making a computer system which will function as a machine which does a process quite equivalent to (though perhaps more crisply defined than) a social process such as trade or endorsement.

We use the applications which tie the SW to what we currently think of as reality for three reasons:

1. We need an interface between the SW and the current social systems that is how the SW system will work at least initially.
2. The social system machine has legislative backing (and public understanding etc.) which we want to exploit;
3. The social system we have works and we only want to change the machine incrementally.

Our reason is not that the current definitions are fundamental or because their specification is inherently beautiful (indeed many existing systems are really crufty). Importantly, we do not define the semantics of something to the real world in such a way as to break the loop, when the loop can be completed in the SW. Here is an example of a loop in the semantic web.

• a. Web server grants access to resource d in response to request is signed with key k1.
• b. key k1 is listed in a [employee list] document signed with k2;
• c. Key k2 is listed in a [w3c member] list signed with k3;
• d. Key k3 is the key with which the web server was set up to trust

This little system can happily run controlling our web site. Now in fact we set it up to model the following social system

• A. A person P1 is allowed to read the member site
• B. The person P1 is an employee of company C2
• C. C2 is a member of the consortium according to Hugo;
• D. Hugo is deemed responsible when it comes to defining member site access.

Now to represent the SW loop a-d is very simple. The conditions can be written in math and proved. The social loop A-D as written is always a rough approximation to the very complex web of trust which is often less dependable than the simpler SW model.

Security has always been plagued by people trying to connect the SW steps (such as a-d) at every stage to the social machine (A-D). For example, this would raises the question of how to identify the person P1 with key k1, introducing the quite unnecessary x.500 directory system which is really not part of the trust loop but becomes a security hole, bringing in unnecessary "trusted" third parties. It drags up endless questions of what "identity" really is anyway. It would raise the question of whether it is Hugo or the webmaster or what that is associated with K3. Before we had finished arguing about identity we would be into arguments about "belief". We would be arguing as to whether Hugo really believes that the person is a member of the company - maybe Hugo does not have to but in his webmaster role he does! These are rat holes. (People don't just belive things to believe to a certain extent, they trust certain source for certain purposes). It would be best to use a different term ("interpretation"?) for the mapping between the semantic and real worlds. (I probably haven't got the philosophical terms right at all and I haven't said "model" once)

So what happens, after we have installed our web server access protocol based on digital signature, is that we then relate things to that. We say that invited experts can get have keys on a given list. The semantic web becomes the definitive machine, and we just have rules at the edges about how it related to things like membership payments. An invited expert becomes defined as someone whose key is on a given list.

What we are looking for from a digital signature spec is the relationship between a signature and a string of bits, and what we are looking for from a semantic web toolbox is the language for writing the conditions a-d. We are NOT looking for either to provide and interpretation language for relating a-d to A-D, ora legal language for writing the steps A-D.

Now, the much-asked question, what is the "semantics" of the digital signature in a-d above? From the SW point of view, those rules are the semantics of the system. The whole thing is self-sufficient from the machine's point of view, except for the edges where the server has to understand what to "give access" is, and where the person has to sign a request or a list. The great thing about the semantic web is that we can make it all work and never actually answer the questions "invited" in what sense? by whom? and Does this mean an invitation which has been accepted? and such other rat holes. We must be careful not to confuse what is said with where it is stored There rare basically four rules which define the access machine. We could store them anywhere. They could be sent in an HTTP request, stored on any number of different web sites, in Java rings and smartcards, send by email or etched in marble. The SW design must not constrain where things are stored.

Where do the "sematics of the signature" lie?

The semantics in the SW are for me the whole loop a-d, which you see, to be a loop, and therefore to allow any processing, must eventually be tried down to the key. When you start to argue something on the basis of a signature by a key, they only next step can be some knowledge about the key. In the semantic web, this is a processing rule about things which are signed with that key. However, that does not mean that the signature has semantics which stored as/with/about the key. In fact, I do not think it is useful to talk about the "semantics of the signature.

Documents have meaning. Signatures by themselves do not.

So it is not useful to ask what the semantics of a signature are. Signatures convey trust, but even that because of a set of statements about keys and documents. There are in society many rules about the trust which is conveyed by the signature under various circumstances. We should not attempt to model those when we make the basic infrastructure of the semantic web.

Abstract: Natural languages, encodings, and similar relationships between one abstract thing and another, are best modeled in RDF as properties. I call these Interpretation properties in that they express the relationship between one value and that value interpreted (or processed in the imagination) in a specific way.

The problem of annotating natural language

There has to date (2000/02) been a consistent muddle in the RDF community about how to represent the natural language of a string. In XML it is simple, because you never have to exactly explain what you mean. You can mark up span of text and declare it to be French.

His name was <html:span xml:lang="fr">Jean-Fran&ccedilla;ois</html:span> but we called him Dan.

Under pressure from the XML community to be standard, the RDF spec included this attribute as the official RDF way to record that a string was in a given language. This was a mistake, as the attribute was thrown into the syntax but not into the model which the spec was defining.

Consider the example in the identity section,

<rdf:description>
   <rdf:type>http://www.people.org/types#person</a>
   <play:name>Ora Yrjö Uolevi Lassila</play:name>
   <play:mailbox resource="mailto:ora.lassila@research.nokia.com"/>
   <play:homePage resource="http://www.w3.org/People/Lassila"/>
</rdf:description>

Now that represents five nodes in the RDF graph: the anonymous node for Ora himself (who has no web address) and the four arcs specifying that this thing is of type person, and has a common name, email address and home page as given.

Where to we add the language property? Of course we could add a language attribute to the XML, but that would be lost on translation into the RDF model: no triple would result.

Attempt 1: a property of the person?

Many specifications such as iCalendar (see my notes@link) would add another property to the definition of the person.

<rdf:description>
   <rdf:type>http://www.people.org/types#person</a>
   <play:name>Ora Yrjö Uolevi Lassila</play:name>
   <play:namelang>fi</play:namelang>
   <play:mailbox>ora.lassila@research.nokia.com</play:mailbox>
   <play:homePage>http://www.w3.org/People/Lassila/</play:homepage>
</rdf:description>

Here, the property play:namelang is defined to mean "A has a name which is in natural language B". In the iCalendar spec, the definition more complex in that the lang property is in same cases the language of a name and in other cases that of the object's description. This is a modeling muddle. The nice thing about doing it this way is that the structure is kept flat, and pre-XML systems such as RFC822 (email etc) headers have a syntax which can only cope with this.

There are many drawbacks to this muddle. Ora may have two names, one in Finish and another in English, and the model fails to be able to express that. Because the attribute is apparently tied to the person and not obviously attached to the name, automatic processing of such a thing is ruled out. Clearly, the structure does not reflect the facts of the case.

Attempt 2: a property of the string?

The second attempt is to make a graph which expresses the language as a property of the string itself. Clearly, "Ora Yrjö Uolevi Lassila" is Finnish, is it not? Yes, Ora is Finnish, but that is different. What we need to say is that the string is in the Finnish language. The problem, then, becomes that RDF does not allow literal text to be the subject of a statement. Never mind, RDF in fact invents the rdf:value property which allows us to specify that a node is really text, but say other things about it too. This is done by introducing an intermediate node.

<rdf:description>
   <rdf:type resource="http://www.people.org/types#person" />
   <play:name rdf:parseType="Resource">
       <rdf:value>Ora Yrjö Uolevi Lassila</rdf:value>
       <play:lang>fi</play:lang>
    </play:name>
   <play:mailbox resource="mailto:ora.lassila@research.nokia.com"/>
   <play:homePage resource="http://www.w3.org/People/Lassila">
</rdf:description>

There we have it, and in an RDF graph at least very pretty it looks. And indeed, we could work with this, apart from the fact that we have made another modeling error. It is not true that the language is a property of the text string. After all, the string "Tim" - is that English (short for Timothy? or French (short for "Timothé")? I don't need to add a long list of text strings which can be interpreted as one language or as another. A system which made the assertion that the string itself was fundamentally English would simply be not representing the case.

Attempt 3: a relationship between them.

In fact, the situation is that Ora's name is a natural language object, which is the interpretation according to Finnish of the string "Ora Yrjö Uolevi Lassila". In other words, Finish the language is the relationship between Ora's name and the string. In RDF, we model a binary relationship with a property.

<rdf:description>
   <rdf:type>http://www.people.org/types#person</a>
   <play:name>
       <lang:fi>Ora Yrjö Uolevi Lassila</lang:fi>
    </play:name>
   <play:mailbox>ora.lassila@research.nokia.com</play:mailbox>
   <play:homePage>http://www.w3.org/People/Lassila/</play:homepage>
</rdf:description>

This works much better. Ora has a name which is the Finnish "Ora". This allows an RDF system to create a node for that string, and a "Finish" link from the concept of Ora the person, maybe a Danish link from the concept of the currency, and an old english link from the concept of weight (1/15 pound), not to mention a Latin link from the concept of the shore.

A problem we may feel is we would like the language to be a string, so that we can reference the ISO spec for all such things, but there is of course no reason why the spec for the lang: space should not reference the same spec.

Another problem we might feel is that it is reasonable for the play:name to expect a string, and in most cases it may get a string: what is the poor system supposed to do in order to accommodate finding a natural language object in place of a string? I guess making a class which includes all strings and all natural language objects is the best way to go. Any use of string which did not allow also such natural language object makes life much more difficult for multilingual software- so this is serious problem.

[[This leads us on to another interesting question of packaging in RDF. There is a requirement in XML packaging and in email packaging and it seems quite similarly in RDF that when you ask me for something of type X I must be able to give you something of type package which happens to include the X you asked for and also some information for your edification. But that is another story.@@@ eleborate and define properties or syntax@@@]]

What is really important is that we are using the ability of RDF to talk about abstract things, just as when we identified people by the resources they were associated with, but avoided pretending that any person had a definitive URI.

Datatypes as interpretation properties^*

Datatypes here I mean in the sense of the atomic types in a programming language, or for example XML Datatypes (XML schema part 2). Defining datatypes involves defining constraints on an input string (for example specifying what a valid date is as a regular expression) and specifying the mathematical abstract individuals which instances of a type represent. One can model the relationship between the representation and the abstract value and the string using a property.

<rdf:Description about="#myshoe">
   <shoe:size>10</shoe:size>
</rdf:Description>

This doesn't tell us what it is 10 of. We could go through life without any model of types: we could define a shoe size as being a decimal string for a number inches. There are many questions and tradeoffs which datatype designers make (for example,

• Can you tell the type of a value from the string representation in every case? (eg 1.4e4 vs 1.4d4 for precision)
• Are the values of different datatypes distinct? (Eg, is 1 = 1.0?)
• Are the set of datatypes extensible? (Eg, can you add complex numbers or prime numbers?)
• Does representation equality imply value equality?
• Does value equality imply representation equality? (Is the only allowed representation the canonical one?)

It would be nice to be able to model these questions in general in the semantic web, in order describe the properties of dat in arbitrary systems. We can introduce interpretation properties which link a string to its decimal interpretation as number, or a length including units. The problem is that the RDF graph which most folks use is the one above. The object of shoe:size is "10".

The simplistic system corresponding exactly to the Attempt 1 above, is to declare that shoe:size is of class integer. This implies (we then say) that any value is a decimal string. Given the string and the type we can conclude the abstract value, the integer ten. This works. It is the system used by XML datatytpes whose answers for the questions above are as I understand it [No, Yes, Yes, Yes, No]. A snag is that you can't compare two values unless you know the datatypes.

To model the representation explicitly in the RDF it seems you have to introduce another node and arc, which is a pain.

<rdf:Description about="#myshoe">
   <shoe:size>
      <rdf:value>10</rdf:value>
   </shoe:size>
</rdf:Description>

We can then define rdf:value to express that there is some datatype relation which relates the size of the shoe to "10". All datatype relations are subProperties of rdf:value with this system. Once it is that form, the datatype information can be added to the graph. You have the choice of asserting that the object is of a given class, and deducing that the datatype relation must be a certain one. You can nest interpretation properties - interpreting a string as a decimal and then as a length in feet. But this is not possible without that extra node. One wonders about radically changing the way all RDF is parsed into triples, so as to introduce the extra abstract node for every literal -- frightful. One wonders about declaring "10" to be a generic resource, an abstraction associated with the set of all things for which "10" is a representation under some datatype relation. This is frightful too you don't have "equals" any more in the sense you used to have it.

Instead of adding an extra arc in series with the original, we can leave all Properties such as shoe:size as being rather vague relations between the shoe and some string representation, and then using a functional property (say rdf:actual) to relate the shoe:size to a (more useful) property whose object is a typed abstract value.

{ <#myshoe> shoe:size "10" } log:implies
{ <#myshoe> [is rdf:actual of shoe:size] [rdf:value "10"] } .

@@@ No clear way forward for describing datatypes in RDF/DAML (2001/1) @@

More examples

Interpretation properties was the name I have arbitrarily chosen for this sort of use. I am not sure whether it is a good word. But I want to encourage their use. Base 64 encoding is another example. It comes up everywhere, but XML Digital Signature is one place.

<rdf:description>
   <play:name parseType="Resource">
      <lang:fi  parseType="Resource">
        <enc:base64>jksdfhher78f8e47fy87eysady87f7sea</enc:base64>
      </lang:fi>
    </play:name>
</rdf:description>

Another example is type coercion. Suppose there is a need to take something of datetime and use it as a date:

<rdf:description>
   <play:event parseType="Resource">
       <play:start parseType="Resource">
          <play:date>2000-01-31 12:00ET</play:date>
       </play:start>
       <play:sumary>The Bryn Poeth Uchaf Folk festival</play:summary>
   </play:event>
</rdf:description>

Such properties often have uniqueness and/or unambiguity properties. enc:base64 for example is clearly a reversible transformation. It it relates two strings, on printable and the other a byte string with no other constraints. The byte string could not in general be represented in an XML document. The definition of enc:base64 is that A when encoded in base 64 yields A. This allows any processor, given B to derive A. The specification of the encoding namespace (here refereed to by prefix enc:) could be that any conforming processor must be able to accept a base64 encoding of a string in any place that a string is acceptable.

Interpretation properties make it clear what is going on. For example,

<rdf:description about="http://www.w3.org/">
   <play:xml-cannonicalized parseType="Resource">
      <enc:hash-sha-1 parseType="Resource">
         <enc:base64>jd8734djr08347jyd4</enc:base64>
      </enc:hash-sha-1>
   </play:xml-cannonicalized>
</rdf:description>

clearly makes a statement, using properties quite independently defined for the various processes, that the base64 encoding of the SHA-1 hash of the canonicalized form of the W3C home page is jd8734djr08347jyd4. Compare this withe the HTTP situation in which the headers cannot be nested, and the encodings and compression and other things applied to the body are mentioned as unordered annotations, and the spec has to provide a way of making the right conclusion about which happened in what order.

Units of Measure (2006)

This pattern applies very well to units of measure.

See, for example a simple ontology http://www.w3.org/2007/ont/unit of units of measure.

Conclusion

Representing the interpretation of one string as an abstract thing can be done easily with RDF properties. This helps make a clean accurate model. However, using the concept for datatypes in RDF is incompatible with RDF as we know it today.

Preface

This personal note I have put into the set of web architectural notes as it expresses fundamental understandings upon which the practical use and power of the web rest.

The questions addressed are about the relationship of the hypertext forms of linked and embedded material to the social concepts involved such as attribution, endorsement, and ownership of information.

Links in hypertext are new in that they can be followed automatically, but the concepts of reference and inclusion of material predate paper. There should not therefore be much confusion about what links imply, but as there have been some strange suggestions recently which would seriously damage the web, I write this note.

Abstract

Normal hypertext links do not of themselves imply that the document linked to is part of, is endorsed by, or endorses, or has related ownership or distribution terms as the document linked from. However, embedding material by reference (sometimes called an embedding form of hypertext link) causes the embedded material to become a part of the embedding document.

Two sorts of link

Basic HTML has three ways of linking to other material on the web: the hypertext link from an anchor (HTML "A" element), the general link with no specific source anchor within the document (HTML "LINK" element) and embedded objects and images (IMG and OBJECT). Let's call A and LINK "normal" links as they are visible to the user as a traversal between two documents. We'll call the thing between a document and an embedded image or object or subdocument "embedding" links.

This distinction is an old one in hypertext. Some systems such Peter Brown's original "Guide" worked only by expanding links inline, and some (such as HTML before the IMG tag was introduced) worked only with normal links.

Normal Links

A normal hypertext link does NOT necessarily imply that

• One document endorses the other; or that
• One document is created by the same person as the other, or that
• One document is to be considered part of another.

Typically when the user of a graphical window-oriented Web browser follows a normal link, a new window is created and the linked document is displayed in it, or the old document is deleted from its window and the linked document displayed in its place. The window system has a user interface metaphor that things in different windows are different objects.

Meaning in content

So the existence of the link itself does not carry meaning. Of course the contents of the linking document can carry meaning, and often does. So, if one writes "See Fred's web pages (link) which are way cool" that is clearly some kind of endorsement. If one writes "We go into this in more detail on our sales brochure (link)" there is an implication of common authorship. If one writes "Fred's message (link) was written out of malice and is a downright lie" one is denigrating (possibly libellously) the linked document. So the content of hypertext documents carry meaning often about the linked document, and one should be responsible about this. In fact, clarifying the relative status of the linked document is often helpful to the reader.

Embedded Material

The relationship between a document and an image embedded in that document is quite different from normal link. (In some designs it is still refered to as a sort of link).

If I say, "To understand this you only have to read this article", or "This is the agreement between us", I am talking about a particular document. It is important that we have a clear picture of what is part of that document and what isn't. Embedded images clearly are part of the embedding document. The author of a document has responsibility for the content, even if the images he or she includes are from another web site.

(There are issues of expectations to be set about availability and security from corruption of remote material, but I do not address these here. Here I just emphasize is that embedded images should be considered part of a document, but documents connected by a normal link should be regarded as separate documents.)

We compose documents out of parts, and the finished work comprises contributions from the parts and also from the arrangement. It is very important that we can include remote parts by reference without having to make a separate local copy. When an embedded image (or sound) is included by reference to its original address (URI) this allows an inquirer to know that address, and hence know the current version of the image. It allows the owner of the image to to a certain extent to know and possibly to control who has access to that image. Also I expect in that in the future it will allow one to find out the owner and licence terms for distribution of that image, which is important for intellectual property rights to be respected on the Web.

Explict distinction

Advertising provides an exception to this rule: a case in which the embedded image is not part of the document.  At risk of making ittoo easy for users to turn  off advertizing, it would be ideal if the distinction were make in the markup between embeeded information which is or is not part of the document.  This would allow, for example, a border to be places around an advertizement to allow the user to realize that it does not come from the same source as the text.  I personally feel that this would be an important step forward in the integrity  of the web. A flag like

<IMG src="banner-ad.gif" foreign>
 

would be fine.

User Interface

When Web documents are presented to people, most current browsers (1997) make a clear distinction between embedded images, which are presented in the same window as the embedding document at the same time, and linked documents which never are. The window system's concept of a "Window" is used to convey when things are part of the same document. It is important for many reasons, some of which were mentioned above, that user interfaces continue to make this distinction.

Frames

The "frames" of HTML unfortunately provide an interface which is less clear. The parts of the document do appear with the same window, but because within a single frame (subsection of a window) one can follow hypertext links replacing content with a separate document, it is easy to create the impression that the owner of the surrounding frames is in fact responsible for the defining document. It is possible that work by the HTML community can produce explict markup (such as the "foreign" flag above) for conveying, when frames are used, which parts of the screen are considered to be the same document. In the mean time, it is appropriate for content providers so make efforts to ensure by the design of (and/or statements on) their web pages that users are not left with the illusion that information within an embedded frame is part of their document when it is really not.

Next: Some dangerous Myths about Links

See Links and Law before reading this.

Linked Data

The Semantic Web isn't just about putting data on the web. It is about making links, so that a person or machine can explore the web of data.  With linked data, when you have some of it, you can find other, related, data.

Like the web of hypertext, the web of data is constructed with documents on the web. However,  unlike the web of hypertext,  where links are relationships anchors in hypertext documents written in HTML, for data they links  between arbitrary things described by RDF,.  The URIs identify any kind of object or  concept.   But for HTML or RDF, the same expectations apply to make the web grow:

1. Use URIs as names for things

2. Use HTTP URIs so that people can look up those names.

3. When someone looks up a URI, provide useful information, using the standards (RDF*, SPARQL)

4. Include links to other URIs. so that they can discover more things.

Simple.  In fact, though, a surprising amount of data isn't linked in 2006, because of problems with one or more of the steps.  This article discusses solutions to these problems, details of implementation, and factors affecting choices about how you publish your data.

The four rules

I'll refer to the steps above as rules, but they are expectations of behavior.  Breaking them does not destroy anything, but misses an opportunity to make  data interconnected.  This in turn limits the ways it can later be reused in unexpected ways.  It is the unexpected re-use of information which is the value added by the web.

The first rule, to identify things with URIs,  is pretty much understood by most people doing semantic web technology.  If it doesn't use the universal URI set of symbols, we don't call it Semantic Web.

The second rule, to use HTTP URIs,  is also widely understood.  The only deviation has been, since the web started,  a constant tendency for people to invent new URI schemes (and sub-schemes within the urn: scheme)  such as LSIDs and handles and XRIs and DOIs and so on, for various reasons.  Typically, these involve not wanting to commit to the established Domain Name System (DNS) for delegation of authority but to construct something under separate control.   Sometimes it has to do with not understanding that HTTP URIs are names (not addresses) and that HTTP name lookup is a complex, powerful and evolving set of standards. This issue discussed at length elsewhere, and time does not allow us to delve into it here. [ @@ref TAG finding, etc])

The third rule, that one should serve information on the web against a URI, is, in 2006, well followed for most ontologies, but, for some reason, not for some major datasets.  One can,  in general,  look up the properties and classes one finds in data, and get information from the RDF, RDFS, and OWL ontologies including the relationships between the terms in the ontology.

The basic format here for RDF/XML, with its popular alternative serialization N3 (or Turtle). Large datasets provide a SPARQL query service, but the basic linked data should br provided as well.

Many research and evaluation projects in the few years of the Semantic Web technologies produced ontologies, and significant data stores, but the data, if available at all, is buried in a zip archive somewhere, rather than being accessible on the web as linked data.  The Biopax project, the CSAktive data on computer science research people and projects were two examples. [The CSAktive data is now (2007) available as linked data]

There is also a large and increasing amount of URIs of non-ontology data which can be looked up.  Semantic wikis are one example. The "Friend of a friend" (FOAF) and Description of a Project (DOAP) ontologies are used to build social networks across the web.    Typical social network portals do not provide links to other sites, nor expose their data in a standard form.

LiveJournal and Opera Community are two portal web sites which do in fact publish their data in RDF on the web.   (Plaxo has a trail scheme, and I'm not sure whether they support knows links). This means that I can write in my FOAF file that I know Håkon Lie by using his URI in the Opera Community data, and a person or machine browsing that data can then follow that link and find all his friends. [Update:] Also, the Opera Community site allows you to register the RDF URI for yourelf on another site. This means that public data about you from different sites can be linked together into one web, and a person or machine starting with your Opera identity can find the others.

The fourth rule, to make links elsewhere,  is necessary to connect the data we have into a web, a serious, unbounded web in which one can find al kinds of things,  just as on the hypertext web we have managed to build.

In hypertext web sites it is considered generally rather bad etiquette not to link to related external material.  The value of your own information is very much a function of what it links to, as well as the inherent value of the information within the web page.  So it is also in the Semantic Web.

So let's look at the ways of linking data, starting with the simplest way of making a link.

Basic web look-up

The simplest way to make linked data is to use, in one file, a URI which points into another.

When you write an RDF file,   say <http://example.org/smith>, then you can use local identifiers within the file, say  #albert, #brian and #carol.  In N3 you might say

<#albert>  fam:child <#brian>, <#carol>.

or in RDF/XML

<rdf:Description about="#albert"
 <fam:child rdf:Resource="#brian">
  <fam:child rdf:Resource="#carol">
</rdf:Description>

The WWW architecture now gives a global identifier  "http://example.org/smith#albert" to Albert.  This is a valuable thing to do, as anyone on the planet can now use that global identifier to refer to Albert and give more information. 

For example, in the document <http://example.org/jones> someone might write:

<#denise>  fam:child <#edwin>, <smith#carol>.

or in RDF/XML

<rdf:Description about="#denise"
 <fam:child rdf:Resource="#edwin">
  <fam:child rdf:Resource="http://example.org/smith#carol">
</rdf:Description>

Clearly it is reasonable for anyone who comes across the identifier 'http://example.org/smith#carol" to:

1. Form the URI of the document by truncating before the hash
2. Access the document to obtain information about #carol

We call this dereferencing the URI.  This is basic semantic web. 

There are several variations.

Variation: URIs without Slashes and HTTP 303

There are some circumstances in which dividing identifiers into documents doesn't work very well.   There may logically be one global symbol per document per document, and there is a reluctance to include a # in the URI such as 

http://wordnet.example.net/antidisesablishmentarianism#word

1. An HTTP GET  request on the URI of the concept returns 303 See Also and gives in the Location: header, the URI of the document.  
2. The document is retrieved as normal

This method has the advantage that URIs can be made up of all forms.  It has the disadvantage that an HTTP request mBrowse-ableust be made for every single one.  In the case of Dublin Core, for example, dc:title and dc:creator etc are in fact served by the same ontology document, but  one does not know until they have each been fetched and returned HTTP redirections.

Variation: FOAF and rdfs:seeAlso

The Friend-Of-A-Friend convention uses a form of data link, but  not using either of the two forms mentioned above.  To refer to another person in a FOAF file, the convention was to give two properties, one pointing to the document they are described in, and the other for identifying them within that document.

<#i>  foaf:knows  [
       foaf:mbox <mailto:joe@example.com>;
        rdfs:seeAlso <http://example.com/foaf/joe> ].

Read, "I know that which has email  joe@example.com and about which more information is in <http://example.com/foafjoe>".

In fact, for privacy, often people don't put their email addresses on the web directly, but in fact put a one-way hash (SHA-1) of their email address and give that. This clever trick allows people who know their email address already to work out that it is the same person, without giving the email away to others.

<#i>  foaf:knows  [
       foaf:mbox_sha1sum "2738167846123764823647";  # @@ dummy
  rdfs:seeAslo <http://example.com/foaf/joe> ].

This linking system was very successful, forming a  growing social network, and dominating, in 2006, the linked data available on the web.

However, the system has the snag that it does not give URIs to people, and so basic links to them cannot be made.

I  recommend (e.g in weblogs on Links on the Semantic Web , Give yourself a URI, and and Backward and Forward links in RDF just as important) that those making a FOAF file give themselves a URI as well as using the FOAF convention.     Similarly, when you refer to a FOAF  file which gives  a URI to a person, use it in your reference to that person, so that clients which just use URIs and don't know about the FOAF convention can follow the link.

Browsable graphs

One important pattern is a set of data which you can explore as you go link by link by fetching data.   Whenever one looks up the URI for a node in the RDF graph, the server returns information about the arcs out of that node, and the arcs in.  In other words, it returns any RDF statements in which the term appears as either subject or object.

Formally,  call a graph G browsable if, for  the URI of any node in G, if I look up that URI I will be returned information which describes the node, where describing a node means:

1. Returning all statements where the node is a subject or object; and
2. Describing all blank nodes attached to the node by one arc.

(The subgraph returned has been referred to as "minimum Spanning Graph (MSG [@@ref] ) or  RDF molecule [@@ref], depending on whether nodes are considered identified if they can be expressed as a path of function, or reverse inverse functional properties. A concise bounded description, which only follows links from subject to object,  does not work.)

In practice, when data is stored in two documents, this means that any RDF statements which relate things in the two files must be repeated in each.  So, for example, in my FOAF page I mention that I am a member of the DIG group, and that information is repeated on the DIG group data. Thus, someone starting from the concept of the group can also find out that I am a member.  In fact, someone who starts off with my URI can find all the people who are in the same group.

Limitations on browseable data

So statements which relate things in the two documents must be repeated in each. This clearly is against the first rule of data storage: don't store the same data in two different places: you will have problems keeping it consistent.  This is indeed an issue with browsable data.   A set of  of completely browsable data with links in both directions has to be completely consistent, and that takes coordination, especially if different authors or different programs are involved.

We can have completely browsable data, however, where it is automatically generated.  The dbview  server, for example,  provides a browsable virtual  documents containing the data from any arbitrary relational database.

When we have a data from multiple sources, then we have compromises.  These are often settled by common sense, asking the question,

"If someone has the URI of that thing, what relationships to what other objects is it useful to know about?"

Sometimes, social questions  determine the answer.  I have links in my FOAF file that I know various people.  They don't generally repeat that information in their FOAF files. Someone may say that they know me, which is an assertion which, in the FOAF convention, is theirs to assert, and the reader's to trust or not.  

Other times, the number of arcs makes it impractical.   A GPS track gives thousands of times at which my latitude, longitude are known. Every person loading my FOAF file can expect to get my business card information, but not all those trackpoints. It is reasonable to have a pointer from the track (or even each point) to the person whose position is represented, but not the other way. 

One pattern is to have links of a certain property in a separate document.   A person's homepage doesn't list all their publications, but instead puts a link to it a separate document listing them.  There is an understanding that foaf:made gives a work of some sort, but foaf:pubs points to a document giving a list of works.  Thus, someone searching for something foaf:made link would do well to follow a foaf:pubs link.  It might be useful to formalize the notion with a statement like

foaf:made  link:listDocumentProperty foaf:pubs.

in one of the ontologies.

Query services

Sometimes the sheer volume of data makes serving it as lots of files possible, but cumbersome for efficient remote queries over the dataset.  In this case, it seems reasonable to provide a SPARQL query service.  To make the data be effectively linked, someone who only has the  URI of something must be able to find their way the SPARQL endpoint. 

Here again the HTTP 303 response can be used, to refer the enquirer to a document with metadata about which query service endpoints can provide what information about which  classes of URIs.

Is your Linked Open Data 5 Star?

Linked Data is defined above. Linked Open Data (LOD) is Linked Data which is released under an open licence, which does not impede its reuse for free. Creative Commons CC-BY is an example open licence, as is the UK's Open Government Licence. Linked Data does not of course in general have to be open -- there is a lot of important use of lnked data internally, and for personal and group-wide data. You can have 5-star Linked Data without it being open. However, if it claims to be Linked Open Data then it does have to be open, to get any star at all.

How well does your data do? You can buy 5 star data mugs, T-shirts and bumper stickers from the W3C shop at cafepress: use them to get your colleages and fellows conference-goers thinking 5 star linked data. (Profits also help W3C :-).

Now in 2010, people have been pressing me, for governmet data, to add a new requirement, and that is there should be metadata about the data itself, and that that metadata should be availble from a major catalog. Any open dataset (or even datasets which are not but should be open) can be regisetreed at ckan.net. Government datasets from the UK and US hsould be regisetred at data.gov.uk or data.gov respectively. Other copuntries I expect to develop their own registries. Yes, there should be metadata about your dataset. That may be the subject of a new note in this series.

Conclusion

Linked data is essential to actually connect the semantic web.  It is quite easy to do with a little thought, and becomes second nature.   Various common sense considerations determine when to make a link and when not to.

The Tabulator client (running in a suitable browser)  allows you to browse linked data using the above conventions, and can be used to check that your linked data works.

References

[Ding2005] Li Ding, et. al.,  Tracking RDF Graph Provenance using RDF Molecules, UMBC Tech Report TR-CS-05-06

Read whole article...

This looks at the Semantic Web design in the light a little reading on formal logic, of the Access Limited Logic system, in particular, and in the light of logical languages in general. A problem here is a that I am no logician, and so I am am having to step like a fascinated reporter into this world of which I do not possess intimate experience.

Introduction

The Semantic Web Toolbox discusses the step from the web as being a repository of flat data without logic to a level at which it is possible to express logic. This is something which knowledge representation systems have been wary of.

The Semantic Web has a different set of goals from most systems of logic. As Crawford and Kuipers put it in [Crawf90],

[...]a knowledge representation system must have the following properties:

1. It must have a reasonably compact syntax.
2. It must have a well defined semantics so that one can say precisely what is being represented.
3. It must have sufficient expressive power to represent human knowledge.
4. It must have an efficient, powerful, and understandable reasoning mechanism
5. It must be usable to build large knowledge bases.

It has proved difficult, however, to achieve the third and fourth properties simultaneously.

The semantic web goal is to be a unifying system which will (like the web for human communication) be as un-restraining as possible so that the complexity of reality can be described. Therefore item 3 becomes essential. This can be achieved by dropping 4 - or the parts of item 4 which conflict with 3, notably a single, efficient reasoning system. The idea is that, within the global semantic web, there will be a subset of the system which will be constrained in specific ways so as to achieve the tractability and efficiency which is no necessary in real applications. However, the semantic web itself will not define a reasoning engine. It will define valid operations, and will require consistency for them. On the semantic web in general, a party must be able to follow a proof of a theorem but is not expected to generate one.

(This fundamental change goals from KR systems to the semantic web is loosely analogous with the goal change from conventional hypertext systems to the original Web design dropping link consistency in favor of expressive flexibility and scalability.The latter did not prevent individual web sites from having a strict hierarchical order or matrix structure, but it did not require it of the web as a whole.)

If there is a semantic web machine, then it is a proof validator, not a theorem prover. It can't find answers, it can't even check that an answer is right, but it can follow a simple explanation that an answer is right. The Semantic Web as a source of data should be fodder for automated reasoning systems of many kinds, but it as such not a reasoning system.

Most knowledge representation systems distinguish between inference "rules" and other believed information. In some cases, this is because the rules (such as substitution in a formula) cannot be written in the language - they are defined outside the language. In fact the set of rules used by the system is often not only formally quite redundant but arbitrary. However, a universal design such as the Semantic Web must be minimalist. We will ask all logical data on the web to be expressed directly or indirectly in terms of the semantic web - a strong demand - so we cannot constrain applications any further. Different machines which use data from the web will use different algorithms, different sets of inference rules. In some cases these will be powerful AI systems and in others they will be simple document conversion systems. The essential this is that the results of either must be provably correct against the same basic minimalist rules. In fact for interchange of proof, the set of rules is an engineering choice.

There are many related ways in which subsystems can be created

• The semantic web language can be subsetted, by the removal of operations and axioms and rules;
• The set of statements may be limited to that from particular documents or web sites;
• The form of formulas used may be constrained, for example using document schemata;
• Application design decisions can be made so as to specifically guarantee tractable results using common reasoning engines.
• Proofs can be constructed by completely hand-built application-specific programs

For example, Access Limited Logic is restricted (as I understand it) to relations r(a,b) available when r is accessed, and uses inference rules which only chain forward along such links. There is also a "partitioning" of the Web by making partitioning the rules in order to limit complexity.

For the semantic web as a whole, then, we do require tractable

• Consistency, that it must not be possible to deduce a contradiction (without having been given one)
• Strength in that all applications must be subsets

Grounding in Reality

Philosophically, the semantic web produces more than a set of rules for manipulation of formulae. It defines documents on the Web has having a socially significant meaning. Therefore it is not simply sufficient to demonstrate that one can constrain the semantic web so as to make it isomorphic to a particular algebra of a given system, but one must ensure that a particular mapping is defined so that the web representation of that particular system conveys is semantics in a way that it can meaningfully be combined with the rest of the semantic web. Electronic commerce needs a solid foundation in this way, and the development of the semantic web is (in 1999) essential to provide a rigid framework in which to define electronic commerce terms, before electronic commerce expands as a mass of vaguely defined semantics and ad hoc syntax which leaves no room for automatic treatment, and in which the court of law rather than a logical derivation settles arguments.

Practically, the meaning of semantic web data is grounded in non-semantic-web applications which are interfaced to the semantic web. For example, currency transfer or ecommerce applications, which accept semantic web input, define for practical purposes what the terms in the currency transfer instrument mean.

Axiomatic Basis

@@I [DanC] think this section is outdated by recent thoughts [2002] on paradox and the excluded middle

To the level of first order logic, we don't really need to pick one set of axioms in that there are equivalent choices which lead to demonstrably the same results.

(A cute one at the propositional logic level seems [Burris, p126] to be Nicod's set in which nand (in XML toolbox <not>..</not> and below [xy]) is the Sheffer (sole) connective and the only rules of inference are substitution and the modus ponens equivelent that from F and [F[G H]] one can deduce H, and the single axiom [[P[QR]][[S[SS]][[UQ][[PU][PU]]]].)

Let us assume the properties of first order logic here.

If we add anything else we have to be careful that it should either be definable in terms of the first order set or that the resulting language is a subset of a well proven logical system -- or else we have a lot of work to do in establishing a new system!

Intractability and Undecidability

These are two goals to which we explicitly do not aspire in the Semantic Web in order to get in return expressive power. (We still require consistency!). The world is full of undecidable statements, and intractable problems. The semantic web has to give the power to express such things.

Crawford and Kuipers The same explain in the introduction their Negation in ALL paper,

"Experience with formally specified knowledge representation systems has revealed a trade-off between the expressive power of knowledge representation systems and their computational complexity. If, for example, a knowledge representation system is as expressive as first order predicate calculus, then the problem of deciding what an agent could logically deduce from its knowledge base is unsolvable"

Do we need in practice to decide what an agent could deduce from its logic base? No, not in general. The agent may have various kinds of reasoning engine, and in practice also various amounts of connectivity, storage space, access to indexes, and processing power which will determine what it will actually deduce. Knowing that a certain algorithm may be nondeterministic polynomial in the size of the entire Web may not be at all helpful, as even linear time would be quite impractical. Practical computability may be assured by topological properties of the web, or the existence of know shortcuts such as precompiled indexes and definitive exclusive lists.

Keeping a language less powerful than first order predicate calculus is quite reasonable within an application, but not for the Web.

Decidability

A dream of logicians in the last century to find languages in which all sentences were either true or false, and provably so. This involved trying to restrict the language so as to avoid the possibility of (for example) self-contradictory statements which can not be categorized as a true or not true.

On the Semantic Web, this looks like a very academic problem, when in fact one anyway operates with a mass of untrustworthy data at any point, and restricts what one uses to a limited subset of the web. Clearly one must not be able to derive a self-contradictory statement, but there is no harm in the language being powerful enough to express it. Indeed, endorsement systems must give us the power to say "that statement is false" and so loops which if believed prove self-contradictory will arise by accident or design. A typical response of a system which finds a self-contradictory statement might be similar to the response to finding a contradiction, for example, to cease to trust information from the same source (or public key).

Reflection: Quoting, Context, and/or Higher Order Logic

@@hmm... better section heading? maybe just quoting, or contexts? one place where we really do seem to need more than HOL is induction.

The fact that there is [Burris p___] "no good set of axioms and rules for higher order logic" is frustrating not only in that it stumps the desire to write common sense mathematically, but also because operations which seem natural for electronic commerce seem at first sight to demand higher order logic. There is also a fundamental niceness to having a system powerful enough to describe its own rules, of course, just as one expects to be able to write a compiler for a programming language in the same language (@@need to study references from Hayes, esp "Tarski's results on meta-descriptions (a consistent language can't be the same expressive power as its own metatheory), Montague's paradox (showing that even quite weak languages can't consistently describe their own semantics)". When Frege tried second-order logic, I understand, Russel showed that his logic was inconsistent. But can we make a language in which is consistent (you can't derive a contradiction from its axioms) and yet allows enough to for example:-

• Model human trust in a realistic way
• Write down the mapping from XML to RDF logic to allow a theorem to be proved from the raw XML (and similarly define the XML syntax in logic to allow a theorem to be proved from the byte stream), and using it;

The sort of rule it is tempting to write is such as to allow the inference of an RDF triple from a message whose semantic content one can algebraically derive that triple.

forall message,t, r, x, y (
  (signed(message,K)
    & derivable(t, message)
    & subject(t, x)
    & predicate(t, r)
    & object(t, y))
   -> r(x,y)
)

(where K is a specific constant public key, and t is a triple)

This breaks the boundary between the premises which deal with the mechanics of the language and the conclusion which is about the subject-matter of the language. Do we really need to do this, or can we get by with several independent levels of machinery, letting one machine prepare a "believable" message stream and parse it into a graph, and then a second machine which shares no knowledge space with the first, do the reasoning on the result? To me this seems hopeless, as one will in practice want to direct the front end's search for new documents from the needs of the reasoning by the back end. But this is all hunch.

Peregrin tries to categorize the needs for and problems with higher order logic (HOL) in [Peregrin]. His description of Henkinian Understanding of HOL in which predicates are are subclass of objects ("individuals") seems to describe my current understanding of the mapping of RDF into logic, with RDF predicates, binary relations, being subclass of RDF nodes. Certainly in RDF the "property" type can be deduced from the use of any URI as a predicate:

forall p,x,y p(x,y) -> type(p, property)

and we assume that the "assert" predicate <rdf:property> is equivalent to the predicate itself.

forall p,x,y assert(p,x,y) <--> p(x,y)

so we are moving between second-order formulation and first-order formulation.

(2000) The experience of the [PCA] work seems to demonstrate that higher order logic is a very realistic way of unifying these systems.

(2001) The treatment of contexts in [CLA] seems consistent with the design we've implemented.

Induction, primitive recursion, and generalizing to infinitely many cases

It seems clear that FOL is insufficient in that some sort of induction seems necessary.

I agree with Tait (Finitism, J. of Philosophy, 1981, 78, 524-546) that PRA is THE NECESSARY AND SUFFICIENT logic for talking about logics and proofs

Robert S. Boyer, 18 Apr 93

also: pra.n3, an N3 transcription of Peter Suber, Recursive Function Theory

also: ACL2: A Precise Description of the ACL2 Logic Kaufmann and Moore 22 Apr 1998, rdf scratchpad entry 26Mar

(for another sort of induction, i.e. as opposed to deduction, see: Circumscription by McCarthy, 1980.)

A quick look at iCalendar

I spent a few hours reading 50 pages of the iCalendar RFC2445 with a view to evaluating proposals to put it into XML. My conclusion early on was that the spec should be written in terms of RDF properties, particularly as it has a clear property/value and parameter/value structure.

Summary

General points I noticed included

1. The spec is full of x-extensions and IANA registries. these would all be done using namespaces in XML
2. There is no summary of properties with their domains and ranges, which would make the spec much clearer.
3. The parameter value type of "URI" implicitly causes dereferencing. This is not clear from the spec but is assumed by the examples.
4. There are a few example of wanton reification, e.g. relationship type.
5. Encodings, for cleanliness: the encoding is a relationship between two objects, not the property of an object. Same comment on XML DSig.
6. I am concerned that I have not found very much protocol defining what how agents interact, or what a message containing a calendar entry means. But maybe that is elsewhere in the spec.

Narrative

When looking for a natural representation of data in a given lanbguage in RDF, one looks at first for the natural structureo fthe language. iCalndar has a nested set of structures which naturally lend themselves to an RDF graph interpretation. Apart from the noted exceptions, this translatoin leads to a set of fairly logically defined RDF properties which could form iCalendar's contribution to the semantic web.

A "calendar" consists of a set of components, such as events, and to-do list and journal entries. These seem natural RDF types. (There is a choice of whether to introduce special a specific property as the relationship between the containing calendar and a specfic type of component, or whther to use generic inclusion property and then specifythe subtype of the component.)

The components have properties, even known as properties in iCalendar. Now each property is in fact a complex thing which has a "value" (implcitly named) and various "parameters" with names.

The named parameters are clearly easily represented as RDF properties.

The values are generally atomic things suhc as integers and strings, with two exceptions. One is when the valeu if the URI and this implies that the actual value is in a document with that URI. Another is that the value datatype "rcecur"is a string which itself has a substructure. This recurrence substructure takes the form of (guess what!) a set of attribute value pairs.

Detailed comments

2.3 Internationalization

If this were XML this would be done for you, with Unicode and the various encodings etc.

4.1 Content Lines

x-name and iana-token are extensions which XML would give for free using namespaces.

"Each property defines the specific ABNF for the parameters allowed on the property"

This makes general parsing impossible, direct conversion into XML difficult. The only hope is that in fact that it not true and there is more consistency than this line leads you to believe! This sounds like a remake of the RFC822 problem which HTTP has in spades: One parser per page of the spec.

4.1.3

Here in the example

ATTACH;FMTTYPE=image/basic;ENCODING=BASE64;VALUE=BINARY:
      MIICajCCAdOgAwIBAgICBEUwDQYJKoZIhvcNAQEEBQAwdzELMAkGA1U
      EBhMCVVMxLDAqBgNVBAoTI05ldHNjYXBlIENvbW11bmljYXRpb25zIE        <...remainder of "BASE64" encoded binary data...>

represents the encoding as though it were a property of the value. It isn't: it is a relationship between the value and thestring expressed here. Nicer to write that.

<attach>
   <fmttype>image/basic</fmttype>
   <base64>MIICajCCAdOgAwIBAgICBEUwDQYJKoZIhvcN
     [...]
   </base64>
</attach>

which would mean (in XML or RDF nonstriped strawman syntax) "Something is attached which has content type image/basic and has base64 encoding MMICCblablahblah".

Note that making base64 a first class relationship (subclass of encoding) makes for brevity and extensibility: with a namespace I can introduce a new one.

Value=binary has all these problems and is unnecessary. It is assumed in base64. The earlier example with the URI

ATTACH:http://xyz.com/public/quarterly-report.doc

has an implicit dereferencing operation which it would be best to expose:

<attach>
   <uri>http://xyz.com/public/quarterly-report.doc
</uri>
</attach>

which means, consistently with the previous example, "something is attached which is identified by URI http://...."

4.2 property parameters

Property parameter values MUST NOT contain a double quote. So I guess that if i want to represent something which does... I attach it?

4.2.1

ALTREP and many of the following parameters can be represented obviously as RDF properties. There needs to be an explicit property between the introduced thing and any "value".

<description>
   <altrep>cid:asdfsadf@sdfsdaf.com</altrep>
   <text>Proext XYZ review meeting</text>
<description>

This becomes more obvious when you look at things like ATTENDEE.

4.2.2.

There seems to be an embryonic notion of type here ("properties with the CAL-ADDRESS value type". I assume this can be formalized. it would be so much simpler if this were tabulated.

4.2.3 Calendar User Type.

"mailto:" is usually in lower case. I thought it was in fact mandatory that it be in lower case.

4.2.5 Delegatees

It is very confusing who ends up being the attendee notionally when both delegates-to and -from are specified. Changing this to RDF, or contemplating doing logical operations on this make one queasy about the solidity here.

ATTENDEE;DELEGATED-TO="mailto:a@y.com";DELEGATED-FROM="mailto:b@y.com":c@y.com

What is that equivalent to? I assume a@y.com goes to the meeting.

4.2.7. See comment about 4.1.3

4.2.9 Free/Busy Time type

make relationships first class

FREEBUSY=FREE: would be better as FREE: to reduce unnecessary complication and allow extension.

If that section of the spec (4.2.9) seems to be self-referential and difficult to read, that is also because it is describing an unnatural part of a clumsy syntax. You don't say "I am free or busy as follows: 12-1pm and we are talking about free here"! because RDF makes these things first class objects and allow you to group FREE and BUSY and REALLYBUSY as subclases of FREEBUSYTYPE life is easier.

4.2.10 language

xml:lang of course is what one would get for free with XML.

4.2.15

"RELATED-TO:RELTYPE=SIBLING" is a classic wanton reification. Just say SIBLING:

Unfortunately the specification defined how calendars can be put into a hierarchical relationship but doesn't say what that relationship *means*. Maybe it does later in the spec.

4.2.18 Sent By

This is a relationship between a mailbox and another mailbox. It is that the owner of one mailbox is being represented by the owner of another. Yes, the message which asserted this data was probably sent by the agent, but the term is misleading when it crops up in the data. This will cause confusion. This is an example of the clarification which arises when you try to represent the meaning of each rdf:property (icalendar:parameter) independently.

4.2.20 Value Data Type

Note that the "URI" data type does not just constrain the value string to be a valid URI, but indicated that the value string is the document you get when you dereference the URI. Big difference, particularly when you automate the base 64 decoding of something.

In general, note XML data types are defined by XML schema working group. See draft @@. A comparison would be a useful exercise.

4.8.4.1 ATTENDEE

"If the LANGUAGE property parameter is specified, the identified language applies to the CN parameter"

That is a terrible bit of design - a typical bit of interference between different headers which is so temping for designers in these flat specs which can't use nesting. How many other clauses like this are there?

LANGUAGE is, I must admit, a problem RDF has a bug with in general. It is difficult to specify that a string has a language without making an intermediate node that you don't want. This is, I realize the same as the intermediate packaging problem: how to let a system know that what it asked for is inside, but in the mean time, here is some useful information about it. Here is a number and by the way it is prime. here is a GIF and by ht way it is copyright. Here is a common name and by the way it is in English. It is interesting to see the way iCalendar has the same problem

4.8.4 UID

There is linking between components of calendars which uses "UIDs" which are mid URIs with the prefix removed. This is a bug

• It removes calendar objects from the URI space so that one cannot refer to them with any other system which uses a URI -- unless you simply assume that you can by using mid:
• The spec is full of recommendations for making identifiers unique.
• It has a given length of 255 characters which is y2k bug asking to happen. Never specify fixed buffer sizes.

4.8.7.4 "SEQUENCE"

This is not in fact a property of an event, but is a property of a given expression of the state of an event. the rule is that it must be incremented by the organizer if the event changes significantly. In a peer-peer world, it is not obvious what to do.

Not reviewed

I skipped most of the rest of the spec but a few very similar concerns arose with some other parts I glanced at.

Conclusion

It seems that RDF nodes for the calendar, for each event etc, and for each icalendar:property is a fairly straightforward mapping.

A spinoff would be a vocabulary which would include useful reusable models of time.The timezone work could be factored out if it is definitive.

Where RDF mapping was not obvious this sometimes coincided with unclear aspects of the specification.

There are three levels at which the RDF mapping could be made

1. A very direct mapping of the ical:properties and parameters onto rdf:properties. Always use the same "value" rdf:property for the VALUE of an ical:property. This would leave some things looking illogical in RDF. It would be simple to define as a mapping, but the definitoin of the properties would be strange in some cases.
2. Make a few simple adjustments to make the RDF more natural. Places to lok for these arehese have been indicated with a @@ in the table. This will make the mapping obvious to an iCal expert reading the RDF, but at the same time make the RDF queries simpler and the properties more reusable. It would move things like RELATED RELTYPE=X into a subclass relationship between X and RELATED which allows generic RDF machinery to process it.
3. An extensive rework in which the logic of rules was largely exposed in RDFS or something stronger would of course be great.

Appendix: Node types

Appendix: rdf:Properties - from "parameters"

Appendix: Calendar component Properties

See spec 4.8

The columns E, T etc indicate whether the subject of the property is permitted to be an event, todo, journal, freebusy, alarm or timezone component.

Examples

@@@

References

There must be a much better list of resources for hacking calendar files of various formats - but until I find it here are some random things I found.

• The iCalendar RFC: RFC2445
• Jetstream: Java classes in Apache's Jetstream which represent the iCalendar properties.
• Open source handheld synchronisation software at openhandheld.org
• PalmOs documentation; file formats (pdf copy)
• Dan Connolly's design research notebook on this

There is a common requirement for the design of a language on the web that it should allow for extensions, but it must allow a clear declaration as to whether understanding of an extension is a requirement to understanding of the document or whether it may be ignored. (See Evolvability)

Historically the lack of such a "mandatory field" has led to a complete inabaility to get any particular guaranteed behaviour be clients on the web.

This is essential for partial understanding and the smooth evolution of the web.

A simple requirement on a language is that it not only provide for its own extension, but provides for a way to explain whether a given extension is optional or not. This is a fundamental key to smooth evolution from the language to a new version.

There are manyways in which it can be done. It can be done term by term, or in bulk about a whole new language. It can be specified in the new document, in the schema for the new language.

XML provides in Namespaces a standard way of extending languages. It should also, in my opinion, provide a standard way to specify mandatopry or optional extensions.

I propose two things:

Sublanguages

The simple assertion that language A is a sublanguage of language B means that the writer's intent is preserved if a dpcument in language A is converted into a document in language B just by relabelling every term as being from langauge B. For XML, this means that a receiver of namespace A can simply process it as though the namespace had been delcared as B.

This assertion has got to be simple enough to put into a document for cases where the functionality is needed without the receiver having to dereference a schema.

Optional/Ignoreable/Mandatory flags for elements

In XML there are three simple thiong you can do with an element you don't understand.

1. Stop, and conclude you do not understand the document, or the clause in the document; Example : logical NOT
2. Ignore the elementand all its contents (including child elements) Example; <Comment>
3. Replace the element with its contents (including children). Example: <bold>

The schema langauge needs to be able to specify these very simply, and indeed it would be neatto be able to do it in a document for a given elemnt, or in one fell swoop for all the elements in a given namespace.

Languages which donot use XML should attend to these needs in their own way!

Grounding the meaning of a document in URI space.

What is the meaning of a document?

The meaning of a document on the Web can be defined more precisely than an arbitrary paper document. Because we have the benefit of a global namespace (URIs), things become possible which were not before. One example is global hypertext; another is the rigid (though rarely absolute) specification of meaning. Just as a hypertext document can now exactly point to another document when it makes a reference (instead of making some vague natural language reference to it), so can a formal document make a precise reference to the language it uses.

A writer of a document uses the language to convey his intent to the reader. It is essential that the intent of the writer can be well defined for both parties and in general for a third party.

The "language" here I means the set of symbols, the syntactic rules which constrain their combination, and some semantics which are conveyed by defining their interpretation in one or more other formal language, or in some natural language.

On the Web, important things are identified by URIs. This should clearly apply both to the document itself and to the language. The party which defines what a URI refers to I call the publisher, or owner of the URI. HTTP allows a delegated system of authority for ownership (DNS) to define ownership of URIs, and it also provides a network protocol to retrieve documents representing that identified by the URI. The text a document is defined by its publisher and the meaning of the language is defined by the publisher of the language.

Natural languages are constantly evolving and rather vague, in that no one (except Scrabble players) use a particular dictionary as a definitive set of meanings. In practice, the meaning of a word in a natural language is the sum of the associations of that word -- logical or poetic -- in the mind of the reader or writer. Of course society works on the basis of a very strong similarity of the webs of association in different people's minds.

In the semantic web, however, meaning is not vague: the idea is that languages must be defined formally and as precisely as possible. The semantic web consists of some "terminal" languages which are defined solely in natural language terms, and some languages for which there are machine-readable interpretations into other formal languages. Whereas programs processing documents in the first sort of language will typically have to be hand coded, documents in the second set may be processed automatically to convert them into languages in the first set.

URIs can be of various sorts, with various properties depending on their scheme (and, for http URIs, the publisher), but some URIs can be dereferenced to a definitive document. The document resulting from dereferencing the URI for a language is a place where the publisher of the language can put definitive information about the meaning of a language.

Language and document subsets

As languages evolve, there can be many languages which are similar. "Similarity" doesn't mean much, but something which is well defined is when a document in one language A can be treated precisely as though it had been in another language B.

Meaning in XML

In XML, a language is a "namespace", and the document about the language is called a "schema". In XML, one document can contain a mixture of languages, and so the schema if written in XML may contain information about syntactic constraints (in XML-schema language) and/or RDF properties (in rdf-schema language), or any combination of the above. (note)

XML puts no constraints on a language apart from syntactic structure. There is not (without RDF and logic or some other higher level) any overall framework into which new languages can be introduced. So, the question of what an XML document means depends first upon the fully qualified name of the document element. No semantics can be attached to any of its descendents in the document tree except in as much as is defined by the specification of that element type in that namespace. One cannot talk about the "meaning" of a subtree of a document without understanding the semantics of the language. In fact, because languages only necessarily define meaning for documents, the only way one can talk about the meaning of a subset of a document is to define a how those parts of the document can be reassembled into a second whole document. This is what must be done when a digital signature is applied to a document.

The Meaning of Digital Signature

The language defines semantics. On the simple philosophy that one place is enough, It is not the place of a digital signature to define semantics. A digital signature on a document may give a party reason to use the information therein for purposes it would not have otherwise. The issuer of a public key may also put constraints on what sort of guarantees are made by signature with a given key. But the signature itself must not affect the semantics - the meaning - of a document. To allow it to would be to create an inconsistency between the intent of the writer of the original document and the meaning of the signed document. So, signatures themselves have no meaning. The meaning has to be ascibed to them by other documents. For example, I may say, "If an organization is a member of W3C according to a document signed with this key, then that organzation is indeed a member". That is a trust statement which gives the key a connection into the world of meaning of documents.

Style as meaning

(Although few people would think of presentation style of a document as its "meaning", and many of us spend a lot of time emphasising the difference between style and content and semantics, in fact much of what applies to style applies to semantics. Therefore the "meaning in terms of presentation" is a good test case for the architecture of the system. (For many documentation systems, the only semantics required is "H2 means a big bold block on the left"!) Style sheets provide an "interpretation"of a document by mapping it onto another well-defined language of formatting properties. The style sheet language gives a good definition (in English) of what is needed. This is an interesting comparison, and I mention it as a place where architectural conssistency should be maintained, but it isn't what I normally mean by "meaning".)

Logical meaning

When XML is used to encode logic, then a document is a formula and the (see Logic on the web). Then, the way new predicates and constants interact is defined by the logic. The way fundamental new parts of the language (such as quantification) are added is part of a more general question of how arbitrary languages interact. Examples we have seen are the mixing of XHTML and XSL. What is the result - XHTML or XSL? A document or a style sheet? Both?

Mixing Languages

XML puts no contarints on a language apart from syntactic structure. There is not (without for example RDF and logic) some overall framework into which new languages can be introduced. This means that every language has to define how it canbe extended by mixing with other languages. Typically it will indicate the element types which can be subclassed by extensions and therefore incorporated into documents wherever that element type is allowed.

One particular example of such a type is common to almost all languages. This is the sentence, the fully qualified assertion or statement, the formula with no free variables. Almost all whole documents count as such, though an interesting counterexample is a style sheet which represent a function: it specified the result document as a functin of an input document, and so itself cannot be said to be a stand-alone statement. (If I sent you a message consisting only af a stylesheet with no coverletter, what would it signify? What would it mean if I digitally signed it?)

With that exception, it clearly makes sense to allow any language which has the concept of a sentence -- maybe any language at all - to allow sentences from other languages to be included anywhere where a sentence of its own could go. This should be a generic feature of XML schemas.

(It is would be against the minimalist principle for XML generically to define other common subclasses. Note that the RDF spec does define properties and node types and the concept of subclassing in RDF. HTML defines things like block and inline elements, which can be subclassed in extensions; SVG and SMIL probably define similar concepts. The significance of this when looking at downloaded support code would be that, for example, in a set of Java classes implementing HTML, that any subclass of "Inline element" would export the same software API to allow it to be justified and line wrapped in a text flow object. So there is a natural correspondence between element type subclassing and support class subclassing, but the tow must remain distinct. Language specifications must always define what a language means without refering to implementations if they can possibly avoid it)

Note that without the assurances given by such information you cannot just go around embedding one language in another. Every language has to address the issue which the concept of RDF transparency potentially solves for RDF. A surrounding XML context must have the ability to quote, deny, negate or whatever any element. In fact, nothing in XML says that the menaing of a fragment is not affected by thing anywhere else in a document. Nothing suggests that the process of removing sub-trees creates a valid document. (How does xml fragment deal with this?)

Grounded documents

We can say a document is "grounded" if its meaning is completely defined because every term used is explicitly, directly or indirectly, an explicit direct or indirect referece to its definition in a document on the Web. Clearly a definition of "grounding" depends on the set of documents one considers acceptable definitions. "Grounded in W3C Recommendations" would imply that the closure under [i.e. set of all the things you can possibly end up with by repeated applications of] the operation of looking up definitions would be a subset of the set of W3C recommendations.

This is the basis for the entire web and internet architecture stack today. (See also: Stack) . All commercial use on the web is largely to be considered in this light, that the meaning of each messaeg sent across the Internet is well-defined by a series of specifications.

(A sense of grounding also can be appliyed seperately to different sorts of "understanding". When "understanding" means presentation to a human for human understanding, a presentation-grounded documents points to all information such as schemata and style sheets which will enable it to be presented.)

Grounding as a myth: the Web of Meaning

The concept of grounded documents is important for predicatble systems, but it is a bad model for the web -- or for life -- in the long run. Words in a natural langauge such as English are not grounded in a unique base set*. Every time you look one up in the dictionary all you find are more words. The world is web-like, and any attempt by the Web to constrain it to be tree-like is bound to force a misrepresentation of realtity. This is the Wittgenstein view of meaning. Understanding this view sometimes confuses people about the very systematic way in which meaning in Internet protocols is defined by layers and layers of specs.

In fact, the two views both apply, one nested inside the other. Yes, meaning is use - but in the Internet protocols, society has set up social constraints - laws and other expectations - which constrain use to be according to the specs. This is a social constraint which your computer is under when you use the Internet, just as when you fill out a tax form you don't have a choice as to how to interpret the meaning of "Adjusted Gross Income on line 39 of a US IRS form 1040". There is a whole department of the government which defines what it is and which socially owns the term. So while the

What will change with the Semantic Web's development is that its grounding in legacy systems will fade into history. Right now, the meaning of "Invoice total vale" is effectively defined by the software which you plug your RDF document into, and how it treats invoices. This is an important way to bootstrap the semantic web with useful terms. That will become less important as many different software poducts share teh same term. In the end, it is weblike form which will characterize the semantic web. Everyone will be defining things in terms of other things which they feel are useful and stable enough. It will be impossible to insist that there be a global ordering between more basic and less basic specifications -- and to do so would stop the web scaling. No one will agree on a directed acyclic graph determining what terms are "more basic" than others. For any set of definitions in one direction, there can always be some reverse definitions which can be seen by others as just as valid.

So, while the concept of documents grounded in a given base set is important for interoperability, it must not be seen as a goal to force the semantic web into an acyclic structure. There will be no single Dewy decimal system for the semantic web. The concepts of well-defined stable specifications will still be essential. So will respect for the definitions of terms. The difference will be that any one will chose their own set of langauges they consider "basic", and find ways of defining other languages they come across in terms of those. A rich web of conversions, translations will grow up to support this. The web of trust will provdie tools for navigating within and selecting from this web in a safe way. And of course, global standarsdw il wlways make like much easier where they can be made.

FAQ: Surely meaning is only defined by use?

This is all very well, runs a popular line, except that to talk about "meaning" at all is basically bogus. The meaning of words, and therefore languages, is defined by use - by how people actuall respond to them, by how they are processed. Surely the only way I can guarantee that someone will interpret a document in a particular way is to have some out-of-band agreement with them first?

Philosophically, it is indeed the case that you need some out-of-band (not in the message itself) agreement. In real life, though, in fact there a lot of widely-held agreements. In fact, the law is a set of agreements which you are deemed to accept whether you formally agree or not. So when you are sent a tax form, you can't argue that the language of the tax form is not one you interpret in that way. they just stick you in jail.

The web works like one big agreement. By connecting your computer to it and getting email from POP and IMAP ports, there is an understanding that what you get are MIME messages, and the same thing when you pick up web page using HTTP. So by using the web you are entering a world where the assumption can be made that messages are to be interpreted by a set of specifications. the specifications are (currently) generally written in english, and imperfect, but basically debate about them is practically about details, not aboutteh philosophy as to whether they apply. So that is why one can in practice talk about meaning.

FAQ: Doesn't the meaning of a document depend on its context?

Of course it does. If i exclose a phtocopy of a document as an attachment, it doesn't mean I am sending you that letter.

However, theer are a lot of contexts for a document which have the same implication for the meaning of that document. Publication, by email to a public list, or HTTP, or FTP, or printing on paper and nailing to a tree, in each case leaves the meaning of a document defined in the same way. These contexts, in which a document is published by a party, or a message converyed from one party to another, are so common and basic that the meaning of the document in these contexts is referred to simply as the meaning of the document (or message).

The webarchitecture separately enumerates the ways in which these contexts actually work under he hood (publication using HTTP, etc) and teh way documents are interpreted and dealt with once published. That way, XML langauegs don't ahve to keep referring to "meaning when received with a 200 code in HTTP".

Preface

This document was written before the Semantic Web Roadmap, but is an introduction to the same ideas. Both introduce the world of machine-readable data on the web. This document introduces the concepts in the historical sequence at W3C, where the first driving applications of semantic web were metadat, and the first driving metadata applications were endorsement labels (PICS).

Documents, Metadata, and Links

The thing which you get when you follow a link, when you de-reference a URI, has a lot of names. Formally we call it a resource. Sometimes it is referred to as a document because many of the things currently on the Web are human readable documents. Sometimes it is referred to as an object when the object is something which is more machine readable in nature or has hidden state. I will use the words document and resource interchangeably in what follows and sometimes may slip into using "object".

One of the characteristics of the World Wide Web is that resources, when you retrieve them, do not stand simply by themselves without explanation, but there is information about the resource. Information about information is generally known as Metadata. Specifically, in the web design,

Definition

The phrase "machine understandable" is key.  We are talking here about information which software agents can use in order to make life easier for us, ensure we obey our principles, the law, check that we can trust what we are doing, and make everything work more smoothly and rapidly. Metadata has well defined semantics and structure.

Metadata was called "Metadata" because it started life, and is currently still chiefly, information about web resources, so data about data.  In the future, when the metadata languages and engines are more developed, it should also form a strong basis for a web of machine understandable information about anything: about the people, things, concepts and ideas.  We keep this fact in our minds in the design, even though the first step is to make a system for information about information.

For an example of metadata, when an object is retrieved using the HTTP protocol, the protocol allows information about its date, its expiry date, its owner, and other arbitrary information to be sent by the server. The world of the World Wide Web is therefore a world of information and some of that information is information about information. In order to have a coherent picture of this, we need a few axioms about metadata. The first axiom is that :

Axiom

That is to say, information about information is to be counted in all respects as information. There are various parts of this.

One is that metadata can be stored regarded as data, it can be stored in a resource. So, one resource may contain information about itself or about another resource. In current practice on the World Wide Web there are three ways in which one gets metadata. The first is the data about a document contained within the document itself, for example in the HEAD part of an HTML documents or within word processor documents. The second is that during the HTTP transfer the server transfers some metadata to the client about the object which is being transferred. This, during an http GET, is transferred from the server to the client and, during a PUT or a POST, is transferred from the client to the server. One of the things which we have to rationalize in our architecture of the World Wide Web is who exactly is making the statement. Whose statement, whose property is that metadata. The third way in which metadata is found is when it is looked up in another document. This practice has not been very common until the PICS initiative was to define label formats specifically for representing information about World Wide Web resources. The PICS architecture specifically allows for PICS labels which are resources about other resources to be buried within the resource itself, to be retrieved as separate resources, or to be passed over during the http transaction. To conclude,

Put another way, metadata can be a first class object.

The second part of the above axiom is:

That is, metadata itself may have attributes such as ownership and an expiry date, and so there is meta-metadata but we don't distinguish many levels, we just say that metadata is data and that from that it follows that it can have other data about itself. This gives the Web a certain consistency.

The Form of Metadata

Metadata consists of assertions about data, and such assertions typically, when represented in computer systems, take the form of a name or type of assertion and a set of parameters, just as in the natural language a sentence takes the form of a verb and a subject, an object and various clauses.

Axiom

This model implies that in general, two assertions about the same resource can stand alone and independently. When they are grouped together in one place, the combined assertion is simply the sum (actually the logical AND) of the independent ones. Therefore (because AND is commutative) collections of assertions are essentially unordered sets. This design decision rules out for example, in simple sets of data, assertions which are somehow cumulative or later ones override earlier ones. Each assertion stands independently of others.

We will see below how logical expressions are formed to combine assertions in more varied ways, and syntactic rules which allow the subject at least of the assertion to be made implicit. But neither of these change the basic operation of combining assertions in unordered AND lists.

Attributes

Assertions about resources are often referred to as attributes of the resource. That is, the type of assertion is an assertion that the object, the resource in question, has a particular named property such as it's author, and in that case the parameter is the name or identity of the author. Similarly, if the attribute is the document's date of expiry then the parameter is that date.

Often, a group of assertions about the same resource occur together, in which case the syntax generally omits the URI of that resource as it is implicit. In these cases, when it is clear from the context about which resource the assertion is being made, the assertion often takes the form of a list of attributes and values. In RFC822 format messages, such as mail messages and HTTP messages, metadata is transferred where the attribute name is an RFC822 header name and the rest of the RFC822 line is the value of the attribute, such as Date: and From: and To: information. The attribute value pair model is that used by most activities defining the semantics of metadata today.

I use the word "assertion" to emphasize the fact that the attribute value pair when it is transferred is a statement made by some party. It does not simply and directly imply that the resource at any given time has that value for the given attribute. It must be seen as a statement by a particular party with or without implicit or explicit guarantees as to validity. Throughout the World Wide Web, as trust becomes an important issue, it will be important for software -- and people -- to keep track of and take into account who said what in terms of data and metadata. So, our model of data of a resource is something about which typically we know the creator or the person responsible, and typically the date of which the information was created, which implies, in the case of a piece of information which makes an assertion, the date at which the assertion was made.

An assertion

(A u1, p, q...)

typically has as explicit parameters,

• the URI of the resource about which the assertion is made (u1).
• some identifier (A) for the type of assertion being made, such as author or date or expiry date.
• other parameters (p, q,...) according to the type of assertion.

As implicit or explicit or implicit parameters,

• The party making the assertion
• The date/time of the assertion
• etc...

We can often make an analogy with programming languages. An assertion in metadata can be compared with a function call in a programing language. In object oriented languages, the object of the function has a special place among the parameters just as the subject of an assertion does in metadata. In object oriented languages, though, the set of possible functions depends on the object, whereas in metadata the set of assertion types is more or less unlimited, defined by independent choice of vocabulary. Anyone can say anything about anything.

A space for attribute names

It is appropriate for the Web architecture to define like this the topology and the general concepts of links and metadata. What about the significance of individual relationships? Sometimes, as above, these are special, defined in the architecture, and having an architectural significance or a significance to the protocols. In other cases, the significance of relationships or indeed of attributes is part of other specifications, other design, or other applications, and must be defined easily by third parties. Therefore, the set of such relationship and attributes names must be extremely easily extensible and therefore extensible in a decentralized manner. This is why

We have already (1997) several vocabularies of attribute names: for example, the HTML elements which can occur within the HEAD element, or as another example, the headers in an HTTP request which specify attributes of the object. These are defined within the scope of particular specifications. There is always pressure to extend these specifications in a flexible way. HTTP header names are generally extended arbitrarily by those doing experiments. The same can also be true of HTML elements and extension mechanisms have been proposed for both. If we look generically at the very wide space of all such metadata attribute names, we find something in which the dictionary would be so large that ad hoc arbitrary extension would be just as chaotic as central registration would be stifling.

Aside: Comparison with Entity-Relationship models.

This architecture, in which the assertion identifier is taken from (basically) URL space differs from the "Entity-relationship" (ER) model and many similar models like it, including most object-oriented programming systems. In an ER model, typically every object is typed and the type of an object defines the attributes can have, and therefore the assertions which are being made about it. Once a person is defined as having a name, address and phone number, then the schema has to be altered or a new derived type of person must be introduced before one can make assertions about the race, color or credit card number of a person. The scope of the attribute name is the entity type, just as in OOP the scope of a method name is an object type (or interface)By contrast, in the web, the hypertext link allows statements of new forms to be made about any object, even though (before anything other than syntax checking) this may lead to nonsense or paradox. One can define a property "coolness" within one's own part of the web, and then make statements about the "coolness" of any object on the web.

This design difference is in essence a resurfacing of the decision to make links mondirectional, sacrificing consistency for scalability.

An advantage of ER systems is that they allow one to work, in the user interface for example, with a set of properties which "should" be defined for each entity. You can define these in the Metadata's predicate calculus by defining an expression for a "well specified" object. ("For all X such that X is a customer X is well-specified if there exists n such that n is the name of X and there exists t such that t is the telephone number of X and...)

end of aside.

Metadata ("Entity") headers in HTTP

In the above it is important to realize that the HTTP headers which contain what can be considered as metadata ("entity headers") should be separated quite distinctly from HTTP headers which do not. HTTP headers which contain metadata contain information which can follow the document around. For example, it is reasonable for a cache to pass such information on without treatment, it is reasonable for clients or other programs which process data to store those headers as metadata with the document for later processing. The content of those headers do not have to be associated with that particular HTTP transaction. By contrast, the RFC822 headers in HTTP which deal specifically with the transaction or deal specifically with the TCP link between the two application programs have a shorter scope and can only be regarded as parameters of the HTTP method. To make this separation clear will be to make it easier not only to understand HTTP and how it should be processed, it will also make it clear which pieces of HTTP can be used easily and transparently by other protocols which may use different methods with different parameters. The clarification of the architecture of HTTP such that both the metadata and the methods can be extended into other domains is an important part of the work of the World Wide Web Consortium. The Internet protocols SMTP and NNTP and HTTP as well as many new and proposed protocols share much of the semantics of the RFC822 headers. Formalizing the shared space and making it clear that there is a single design for a particular header, rather than four designs which are independent and happen to look very similar, requires a general architecture, some careful thought, and is essential for the future design of protocols. It will allow protocol design to happen in small groups which can take for granted the bulk of previous work and concentrate on independent new design.

Authorship of HTTP entity headers

It may be possible to remove or at least encompass the apparent anomaly of metadata transferred from an HTTP server by creating a special link type which links the document itself to the set of attributes which the server would give in the HTTP headers. In other words, the server would be able to say, "here is a document, here is some metadata about it, and the metadata about it has the following URL". This would allow one, for example, request a signed copy of the HTTP headers. It would allow one to ask about the intellectual property rights of those headers, and the authorship of those headers.

It is important to be completely clear about the authorship of the HTTP headers. The server should be seen as a software agent acting on behalf of a party which is the publisher or document author: the definer of the URI to resource identity mapping. The webmaster is only an administrator who is responsible for ensuing that (through an appropriately configured server) the transactions on the wire faithfully represent the statements and wishes of that party.

Links

An assertion of relationship between two resources is known as a link.

In this case, it is a triple

(A u1 u2)

of:

• the type of assertion being made, that is, the relationship which is being asserted,
• the first URI,
• and the second URI.

These sorts of assertions, links, are the basis of navigation in the World Wide Web; they can be used for building structure within the World Wide Web and also for creating a semantic Web which can express knowledge about the world itself. That is to say, links may be used both for the structure of data, in which case they are metadata, but also they may be used as a form of data.

Links, like all metadata can be transferred in three ways. They can be embedded in a document, which is one end of the link, they can be transferred in an HTTP message, for example what is called the header of the document, and they can be stored in a third document. This latter method has not been used widely on the World Wide Web to date.

Goal: Self-describing information

A critical part of the design of the whole system is the way that the semantics of metadata or indeed of data are defined. The semantics of metadata in our RFC822 headers in mail messages and in http messages are defined by hand in english in the specifications of those protocols. The PICS system takes this to one stage further in terms of flexibility by allowing a message to contain a pointer to the document which defines, in human readable terms, the semantics of each assertion made within a PICS label. In the future we would like to move toward a state in which any metadata or eventually any form of machine readable data carries a reference to the specification of the semantics of all the assertions made within it.

For example, suppose that when a link is defined between two documents, the relationship which is being asserted is defined in a such way that it can be looked up on the World Wide Web (i.e. using some form of URI), and someone or some program, which has not come across that relationship before can follow the link and extend its understanding or functionality to take advantage of this new form of assertion.

In the case of PICS, one can dynamically pick up a human readable definition of what that assertion really means. In PICS (and in theory in SGML using DTDs), one can also pick up a machine readable definition of what form that assertion can take, what syntax, what types of parameters it can take. This allows a human interface to a new PICS scheme to built on the fly. To go one step further, one could, given a suitable logic or knowledge representation language, pick up a machine readable definition of the semantics of that assertion in terms of other relationships.

The advantages of such self describing information is that it allows development of new applications and new functionality independently by many groups across the web. Without self-describing information, development must wait for large companies or standards committees to meet and agree on the commonly agreed semantics.

Of course a pragmatic way of extending software to handle new forms of information is to dynamically download the code to support a software object which can handle such data for one. Whereas this is a powerful technique, and one which will be used increasingly, it is not sufficient. It is not sufficient because one has to trust the implementation of the object, and the state.

Goal

Building Applications using Link Relationships

It turns out that a very large number of applications both built on top of the web and also built within the infrastructure of the Web can largely be built by defining new relationship types. Examples of these are the document versioning problem which can be largely solved by defining link values relating documents to previous and future versions and to lists of versions; intellectual property rights, distribution terms, and other labeling which can be solved by making a link from one document to the document containing the metadata.

The web is a very general concept -- one universal space of information. The concepts it requires such as identifiers and information resources (documents) are as general and abstract as possible. However, there have been some design decisions made which define some interfaces, and effectively define modules or agents which are independent. These agents are independent in many ways

• There is knowledge they have individually but do not share
• There is knowledge their designers had individually but did not share

This is basic modularity. The interfaces are defined by the data formats and protocols, and the important features to understand about the design I have ranted about in the linked articles in this series. This modularity, ability for different parts of the system, shows up when different specs are independent, such that you could change one without having to change the other.

The Information Resource

(Formerly, Resource)

This is the current term for a certain unit of information in the Web. In many cases on the current Web, thinking "document" will do. It is something which conveys information. The Web model is that information in the information space is in the abstract chunked into addressable things known as resources.

In the technical architecture, resources have identifiers, Universal Resource Identifiers, and the properties of these identifiers are elaborated later. In fact the concept of a unit of information is central, not only in the technical architecture, but in society's concepts of information, as a document is not only the unit for reference, retrieval and presentation (typically), but also the unit of ownership, license to use, payment, confidentiality, endorsement, etc. So though technically we can derive such things as compound document, generic documents, and resources which look anything but the typical notion of a "document", we have to be able to support these social aspects of information at the same time, so we can't mess with it too much.

Fragment Id and "#"

In the hypertext architecture, when making a reference, such as a hypertext link, we don't just refer to an information resource. Well, we can, but we can also refer to a particular part of or view of a resource. The string which, within the document, defines the other end of the link has two parts. It has the identifier of the document as a whole, and then optionally it has a hash sign "#" and a string representing the view of the object required.  This suffix is called a fragment identifier.  (Even though it doesn't represent necessarily a fragment of the document: it could represent how the document should be viewed.). The fragment identifier only has relevance in the context of the web page in question. This has an implication how the software is built. For example, An "access" module can be given just the bit of the URI without the fragment identifier. It gets the information, and creates a software object for the hypertext page. That object is passed the fragment identifier.

In fact, analyzing the system a little more, the access function can be broken into the underlying access which creates the object by passing two things to some kind of object creator ("factory"): a data stream and a MIME type.

Generally

Hypertext is a specific application, but this principle works for other applications on the Web. In fact, when we discuss webizing an application, we take some computer language, and we take what were document-global things, say global variables in a programming language, and make them truly global by appending the URI of the document and "#".

Clearly, in different applications the fragment identifier will have completely different function. The independence here means that new applications (such as the Semantic Web) can be built, just like hypertext web, just by introducing new types of document.

Independence

The model of how the web works is that there are two separate functions.  The part (blue in the picture) which accesses the document deals with its identifier, but does not know what view will be required.  It creates some software object which represents and presents the resource. That object does not need to know how it was created (necessarily), and so does not need to know the URI it was identified by. However, it does know how to interpret the Fragment ID.

So we have two axioms:

The equivalent axioms when we are talking about specifications amount to:

Why?

For one thing, consider the special case of a link within a document.  In this case, the link only specifies a fragment identifier.  The object can follow the link itself.  It doesn't have to consult the access code in order to figure out  where the link goes to.  Because the "#" syntax s universal to all access methods, the object can process the link internally.  For a static HTML file, for example, this means that you can write and HTMl file with internal links without worrying or knowing about exactly what URIs the file will get.  It means you don't have to alter the file if you chose to serve it in some new name or address space.  If the "#" syntax was not a universal specification for the web, this would break: you couldn't do it. As Jim Gettys points out, as the era of digitally signed documents comes upon us, changing a signed document will break the signature on it. So allowing one to make a self-consistent document with internal links in a way independent of the namespace is even more essential.

Why else?

This independence is very important for the evolution of the Web.  It means that people can go off and design all kinds of new systems for naming, addressing and accessing documents, without having to worry about what sort of documents will be moved.  It means that people can go off and make new media types (MIME types), each of which can have different concepts for views and fragments, without having to talk to the people developing the access technology. This has already (1998) proved incredibly enabling to the community, as HTTP has advanced in parallel with many other ways of accessing data, and the number of exciting media types has grown very rapidly, and will be the key to many new revolutions built on top of the basic Web idea. 

If you look at the diagram you ill notice how the fragment IDs are generated by and understood by just the one module.  You see how, when designing a new MIME type, one is quite free to be creative in making new and powerful forms of fragment ID, knowing hat no other specifications will refer to them, and nothing else will break.

Document sets and relative addressing

Now let us look at what happens when we follow a link.  For example, say a hypertext page is clicked on.  The page has a representation of the end point of the link.  It hands it to the application.  In fact, often, there are links between pages whose URIs are very similar and only differ in the right hand part.  This isn't true of all name spaces: for example, when making links between news articles identifies by the news id (news:foo) unique ID, you have to specify the whole thing. However, if you restrict publication of a set of documents to a hierarchical name or address space, then you can arrange for documents which are very related and have many links to be in the same part of the tree.

In this case, the links between these documents are "relative URIs".

What happens then is that the relative URI, which only has the locally different part of the URI in it, is handed back to what in the diagram I have called the "application", to be turned into an absolute URI by being combined with the absolute URI of the resource, which the application has remembered.

Note that the application is aware of the absolute URI but still the resource does not have to.

Note that the fragment id is still circulated around a loop between the object (green) which understands it and the applications (yellow) which handles it transparently but does not understand or change it.

Now there was a design decision that the application could have passed to the access module both the relative URI and the absolute URI. Then, different namespaces would have been able to have different algorithms for resolving a base URI and a relative URI into a new absolute URI. But the decision was made that the relative address format should be common across all name spaces.

Why?

Just as we considered internal links above, now consider relative links between a bunch of documents, like the sections of a book, which are close in the tree.  In practice, such document sets are moved from place to place, from file systems into HTTP space or FTP space, and because the relative address rules are universal, the documents do not have to be modified every time they are moved. (Yes, if you move half the set to one place and half to another, you have to fix links).  This is happening all the time.  People are creating and programs are generating hypertext with relative links without knowing or caring what absolute URI will be used to refer to the material.

The access scheme

The so-called "access scheme" is the first part of the URI. As we have seen above, you don't have to know anything about it to parse relative URIs or to process the fragment identifier of a URI. The knowledge of particular schemes is limited to the "access" function (blue in the above diagram).

The scheme is a very important flexibility point, and should not be abused. Anyone dereferencing a URI must have a knowledge of the scheme it uses.

The access scheme defines a huge part of URI space. The scheme defines a subspace with particular properties

The access scheme is by definition the highest point of flexibility. What does that mean? It means that if the whole Web develops problems which we cannot solve within the existing protocols, or if new spaces are designed which really can't be accessed through or mapped into existing spaces, then we can create a new space. We have faith that we will be able to use this flexibility point in the future, because it worked successfully for integrating the older spaces such as Gopher and FTP spaces into the Web.

The danger of too many access schemes

However, we do not do this lightly. When we introduce a new space, it may have very different properties and we expect that the deployment of new software will be needed to allow access to it. Some spaces may be gatewayable into HTTP space, and this will often provide a transition path. This is why early browsers allowed one to declare in a configuration file what gateways to use for what new spaces.

If we use this extension point frivolously, ironically, it will cease to work. Suppose very many schemes are introduced. The access scheme space itself becomes a namespace with all the problems which current namespaces such as DNS are trying to solve, but which are very hard problems:

• Clashes in the namespace would destroy interoperability;
• Ownership of the space becomes commercially valuable;
• Democratic and fair management becomes essential and difficult;

Worse, though, technology will be needed to automatically dereference the schemes themselves and download code to handle them. Something like DNS will be needed. The top level namespace then becomes in fact DNS, or something like it. This, however, begs the question. What happens if later DNS needs to be replaced? There is no top-level extension switch left. The world is stuck with whatever form of access-scheme name service exists.

Therefore, I conclude that access schemes should not be open to trivial extension, and that the access scheme should only be extended by the introduction of new standards with full open review by the entire community.

Alternatives to new schemes

Whereas some schemes (like "data:") are clearly neat and new and orthogonal to HTTP, many schemes could in fact be integrated into http, using HTTP extension mechanisms.

In fact, is HTTP is to be taken as a general computing protocol, then use of an extensible language system for the HTTP request message would allow a huge amount of extension, covering protocols with different functionality (exporting different interfaces).

Evolving scheme spaces

When considering the evolution of a space, it is important to remember that primarily the access scheme refers to a part of the URI space, and secondarily it refers to a protocol. Therefore, one can in fact change the protocols used to access resources within a scheme's namespace, without changing the space. For example, a new DNS protocol could be introduced which over time would replace the current one, without changing the DNS space. This would effectively redefine the HTTP and FTP protocols, but would not harm the namespaces. When touch-tone dialing was introduced, the telephone numbering system remained the same. So an indexing system could be introduced which, when deployed, would allow http:// space objects to be found with greater reliability or speed than the current protocols, while maintaining the HTTP space as being the concatenation of a DNS name and an opaque string.

Simple things make firm foundations

You can look at the development of web technology in many ways, but one way is as a major software project. In software projects, the independence of specs, has always been really important, I have felt. A classic example is the independence of the HTTP and HTML specifications: you can introduce many forms of new markup language to the web through the MIME Content-Type system, without changing HTTP at all.

The modularity of HTML itself has been discussed recently, for example by Ian Hickson, co-Editor of HTML5:

Note that it really isn't that easy. For example, the HTML parsing rules are deeply integrated with the handling of <script> elements, due to document.write(), and also are deeply integrated with the definition of innerHTML. Scripting, in turn, is deeply related to the concept of scripting contexts, which depends directly on the definition of the Window object and browsing contexts, which, in turn, are deeply linked with session history and the History object (which depends on the Location object) and with arbitrary browsing context navigation (which is related to hyperlinks and image maps) and its related algorithms (namely content sniffing and encoding detection, which, to complete the circle, is part of the HTML parsing algorithm). - Brainstorming test cases, issues and goals, etc., Ian Hickson

and in reply by Laurens Holst:

I don't know the spec well enough to answer that question, but I'd say modularization (if I may call it so) would make it both easier to grasp as individual chunks, for both the reviewing process and the implementing process. brainstorming: test cases, issues, goals, etc.. - Laurens Holst

The <canvas> element introduces a complex 2D drawing API different in nature from the other interfaces, which concentrate on setting and retrieving values in the markup itself; the client-side database storage section of the specification is another such interface. While the <canvas> element has a place in the specification, the drawing API should be defined in a separate document. Hixie expressed a similar sentiment (and see the group's issues about scope):

The actual 2D graphics context APIs probably should be split out on the long term, like many other parts of the spec. On the short term, if anyone actually is willing to edit this as a separate spec, there are much higher priority items that need splitting out and editing...

It would also be nice if the <canvas> element and the SVG elements which embed in HTML did so in just the same way, in terms of the context (style, etc.) which is passed (or not passed) across the interface, in terms of the things an implementer has to learn about, and things which users have to learn about. So that <canvas> and SVG can be perhaps extended to include say 3D virtual reality later, and so that all of these can be plugged into other languages just as they are plugged into HTML.

There are lots of reasons for modularity. The basic one is that one module can evolve or be replaced without affecting the others. If the interfaces are clean, and there are no side effects, then a developer can redesign a module without having to deeply understand the neighboring modules.

The flip side is that a cleanly designed module designed as part of one system can be re-used in other systems.

It is the independence of the technology which is important. This doesn't, of course, have to directly align with the boundaries of documents, but equally obviously it makes sense to have the different technologies in different documents so that they can be reviewed, edited, and implemented by different people.

The web architecture should not be seen as a finished product, not as the final application. We must design for new applications to be built on top of it. There will be more modules to come, which we cannot imagine now. The Internet transport layer folks might regard the Web as an application of the Net, as it is, but always the Web design should be to make a continuing series of platforms each based on the last. This works well when each layer provides a simple interface to the next. The IP is simple, and so TCP can be powerfully built on top of it. The TCP layer has a simple byte stream interface, and so powerful; protocols like HTTP can be built on top of it. The HTTP layer provides, basically, a simple mapping of URIs to representations: data and the metadata you need to interpret it. That mapping, which is the core of Web architecture, provides a simple interface on top of which a variety of systems -- hypertext, data, scripting and so on -- can be built.

So we should always be looking to make a clean system with an interface ready to be used by a system which hasn't yet been invented. We should expect there to be many developers to come who will want to use the platform without looking under the hood. Clean interfaces give you invariants, which developers use as foundations of the next layer. Messy interfaces introduce complexity which we may later regret.

Let us try, as we make new technology, or plan a path for old technology, always to keep things as clean as we can.

In this article:

1. Syntax for Graph traversal ("paths")
2. Infix operators
3. Syntax for sets
4. Other issues

Syntax for graph traversal

There is a strong need for a neat syntax for converting an expression for x into an expression for something removed one step along the graph from x by an arc of type (rdf:Property) p. For example, if x is a person then we want an expression for x's email address. (I am dropping the prefixes in this discussion to reduce clutter)

"Neat"? Compact, powerful, simple, naturally understandable because of metaphors with existing use of similar syntax.

Strictly, we are talking about some y, such that p(x,y), or in n3, [is p of x]. There is no implication in this syntax at the moment (but could be later) that there is only one such y. The information that there can be only one such y, when it is so, is conventionally in stored by noting that p is a daml:uniqueProperty property. This can be stated in any document, though current colloquial use puts it into the schema for p.

I will call moving from x to [ is p of x] forward traversal, and moving from x to [p x] backward traversal. My instinct is that forward traversal, which is the only thing you can do naturally in many systems of linked objects, is more common need in the language than backward traversal.

Backward traversal can also be expressed as forward traversal through the inverse of a property, so a compact expression for the inverse of a property would be an alternative, so long it was clear when this was syntactic device for making a backward link, and when(if ever) it was actually used to make

We need both to chose punctuation and also the grammar, as to the precedence of the operator if any. To be able to write "the person whose wife's uncle is driving my bother's car" . Mostly here I am looking at traversal expressions going left to right with no precedence, but "of" as used in english is an exception in that it is right to left.

Use case examples

Forward traversal: The phone number of the home of the chair of the conference x,

The last case, labelled traversal, is in fact much more than graph traversal - by embedding variables into the graph in search template (rule antecedent), one make a reference which can be used in a rule conclusion. One can also, by reusing a variable more than once, make multiply connected (right phrase?) graph in place of a tree.

Dot

This problem has strong analogy with moving from an object to a slot in an object. Python, c++, etc: x.email, so that metaphor is a natural one to pick up.

Pro: For programmers, this is a natural.

Con: Dot as the end of an n3 sentence would have to be protected by following space or punctuation. The language is made more complex in that either some tricky tokenizing with some form of look-ahead becomes necessary.

There is no equivalent convention as far as I know for backward traversal, so let's pick something random and inverse to "." -- say "^". (Metaphor: back up rather than down forwards?). Think of "^" as a combination of "." and an operator to generate the inverse property. (Or maybe "^" should be that property, which would make foo.^bar a back-traversal except that it would actually be represented using an extra triple.)

Bang

There is a form of path familiar to those who knew email and net news in the days of source routing: when one had to specify a series of machine names through which the mail had to be forwarded, as in mcvax!cernvax!online!timbl. Though few current users will remember it, it has the advantage over dot of being unused elsewhere in teh N3 syntax. This leaves the N3 language simpler.

Multiply, Divide

Metaphor: Units of measure

A snappy syntax is useful in the leaves of an expression tree,. Examples come up frequently when the logical way to express data types, units of measure, and so on is with a graph traversal. With units of measure, people use use multiplication and division, and these actually make sense mathematically.

Cost = 100*dollars or even Cost/dollars = 100 and Cost/day=100*dollars.

Pro: / and * are indeed inverse, when you have unique and unambiguous functions: x/y*y =x.

Con: This is not always the case! Also, "*" and "/" in math, and in units of measure, have properties like commutativity which you expect of "*" and it doesn't have in this context/. Also, I had expected that it would be pragmatic to add in operators directly to the syntax for convenience, and so was reserving + - * /.

Keywords - which, of, 's, the

The english language suggests some keywords.

"which" I have considered using in a sentence to turn the current object into the new subject. There are two forms I had thought of, I'll call them "which" and "thatwhich" for now. "Which", as in english, applies to a started object and allows labelled traversal. "thatWhich" is used for backward traversal, though the grammar is different.

:joe :son :johnny which has :girlfriend :jane.

:joe :son thatWhich :girlfriend :jane.

thatwhich has :home thatwhich has :email thatwhich has

Pro: which reads very well (unless you insist on whose!), especially with N3's optional has before the property.

Con: thatwhich is unbeliveably ugly. Even which, while reading well, is not a very concise form.

A possibility is to just use which, with [] for the that or something which precedes it in english grammar. In fact, if someone wants something as a synonym of [] I wouldn't violently object.

:joe :son [] which :girlfriend :jane.

A synonym for "which" could be the more mathematical "suchthat", which suggests a vertical bar.

:joe :son [] | :girlfriend :jane.

This makes an effective traversal operator []| which is an eyeful, but the pipe is nice as a connector.

joe son :johnny | girlfriend jane | mother [] | email <audey@example.com>.

"Of" is interesting, though could be confusing that it parses right to left

email of home of boss of x means email of (home of (boss of x))

I just noticed that when I write on the blackboard, % and of look pretty similar, so % to be read as of would a possibility for forward traversal prefix operator.

The astute will have noticed that "of" is already used as a keyword in N3. However, all is not lost, in fact much could be gained. Could one not split "of" and "is" into separate features of the language, p of y being simply short for what is currently [ is p of y], and is being an operator which at the syntactic level indicates that two things are the same node.

(This is not the same as N3's =, which is daml:equivalentTo, which has axioms about properties of similar things being the same, but is not involved at this level. N3 and RDF treat different URI-identified nodes separately, whether or not a daml:equivalentTo arc joins them))

This allows things like

joe brother [ is fred; wife margy; kids jane, john].

Contrast "of" with with the english 's, German -es

x's boss's home's email meaning ((x's boss)'s home)'s email

which reminds one of Ada's

x'boss'home'email

of whose etymology I am unaware.

Con: I was kinda thinking of keeping all the quotes I can in hand for use in various forms of quotation! So many languages needs many forms of quotation and run out of options all to fast. (XML an Python both use " and ' to mean the same - a waste if you ask me!)

One could go the other way and just use a keyword "s"

x s boss s home s email.

or use a "$" with a closeness to "'s" and expectation of being read aloud as such:

x$boss$home$email

"The" in english signifies the uniqueness of something, and so could be used to indicate that something is indeed a function.

the email of the home of the boss of x

Arrows

Access limited logic, and the original N3 design, one of the conceptual graph serializations, and other languages derived from a transcription of whiteboard circles-and-arrows diagrams, use "->" or ">" as a traversal operator. Multics used (I understand) ">" for descent of a directory tree and "<" for ascent, so ../../foo/test would be <<foo>test which is neat even though it frightens the xml-minded side of one.

N3 uses <> to surround URIs, which i suppose could be changed, but it interferes strongly with this.

Slashes

Same idea as arrows, but using slash.

Pro: The metaphor with directory traversal is useful (even though the graph being traversed is not always a tree).

Pro: A nice simplicity.

x.uncle^boss.home.email

becomes

x/uncle\boss/home/email

Con: Unix types could find it strange when finding their universal escaping character used as anything else. And it rules our using it for that form.

Con: The confusion which Microsoft introduce by using backslashes for directories has done lasting harm to the community, leaving many people still unsure which is which. This sort of

Parens

The application of a monadic function is a special case of the traversal of a graph arc, so syntactic metaphors from functions would seem appropriate. The most obvious case is when a function takes a list, to just abut the function identifier to the list, looking like a regular function call in more languages than I could name:

x = math:max(y z w) for x = [ is math:max of (y z w)]

Pro: Looks great.

Con: Doesn't work when the function doesn't take a list. Also, if you get a space in between, it means something completely different. Hopefully it will in some cases at least be a syntax error, but not within in a list.

Maybe a separator of some sort as punctuation would work a left/right reversed from of "."

x = math:max$(y z w)

Summarizing

One thing that becomes evident: it can be really difficult to read the backward traversal in english. Like many systems, (including WWW) , english is optimized for forward traversal

Swan

Sandro's swan language used a name immediately followed by "(" as a function opener as in sum(2 3).

He also used "." for path traversal.

Infix operators

I had reserved * / + - for infix operators for arithmetic. The | operator for or and & for and (or union and intersection of sets) are also reasonable to use in this way.

If N3 is to have a to have a path toward becoming a language in which arithmetic and set operations are easy to write, it is hard to improve on infix notation. This would, however, change the form of the language significantly. It isn't clear that it would still be predictively parsable.

Sets

The following considers design alternatives in extending N3 to include a notation for set literals. 2005/1/1

Background on containers

In the area of containers, RDF started with some "Sequences" and "Bags" which were in my opinion and with the benefit of hindsight, sub-optimal (The infinite rdf:_1 series of predicates was downright weird, and taking it into consideration made code much mroe complicated. Futher, for all the arbitrary complexity of the rdf:_nnn predictaes, they didn't tell you that essential bit of information as to when the container was finished: what wasn't in the container) .

RDF does however have a collection which is an ordered list, and is very useful. N3 has a shorthand syntax ( 1 2 3 ) for the list of the numbers 1, 2 and 3, and the RDF/XML syntax has parseType="collection" shorthand. There is also defined a way of expressing lists in triples using blank nodes, using rdf:first and rdf:rest, and rdf:nil. The list 1 2 3 would be expressed as

[ rdf:first 1; rdf:rest [
     rdf:first 2; rdf:rest [
        rdf:first 3; rdf:rest rdf:nil]]]

This is, if you like, a reification of a list. It described it totally. Some RDF systems actually store lists in this way. The RDF and OWL specs together are not (as far as I was aware) very clear about the axioms of lists. One would expect clear axioms that all lists exist, that any two lists with the same first and rest are owl:equivalent, and so on.

Introducing sets

It turns out that in many cases in applications we have seen, containers are in fact logically unordered sets, not ordered lists. Whether it is mail addresses on a mailing list, or rows in a database, or statements in an N3 formula, the order is immaterial, and something can occur in the set once or not at all.

In these circumstances to use a list to represent the data is suboptimal in may ways. For example,

• It is not clear when two different lists actually have the same members in a different order that they represent the same set;
• The information about what is in fact a set end up being communicated out of band, or just assuemd by those who know the application;
• Underlying implementations cannot use code library support which is optimized for sets.

For these reasons it is useful to have sets in the language in the same way as lists: to have a reification - a way of expressing them in triples so as to be able to pass them though general RDF applications whcih may be unaware of them, and a shorthand syntax to allow them to be written effeciently.

Reification

It turns out that OWL provides is with owl:oneOf, a relationship between a class and a list, such that the class is the class of things which are members of the list. Unless for some reason one wants to make sets different from classes, it seems appropriate to use classes for sets, and furthermore to use owl:oneOf as the constructor which allows us to specify a specific set in terms of an arbotrary ordering of its contents. The set of numbers 1,2 and 3 would then be written as

[ owl:oneOf (1 2 3)] 

or, to elaborate it down to triples:

[ owl:oneOf 
  [ rdf:first 1; rdf:rest [
     rdf:first 2; rdf:rest [
        rdf:first 3; rdf:rest rdf:nil]]]]

Of course, any reification of a set whish lists the same members in a different order describes the same set.

Syntax

This is the more difficult choice! Here is a table of suggested syntax extensions to N3 for sets.

The current choice is {$ 1, 2, 3 $} which is conventional mathematical set notiation, plus dollar signs to distinguish a set from a formula.

An interetsing possibility pointed out by Sandro Hawke is to actually make sets and formulas examples of the same thing. A formula is just a set: a set of statements. This makes statements first class objects. This is inherently appealing in its symmetry. However, as there is no statment opener syntax, only the closer (".", and effectively ";" and ","), there is no way for the parser to know in advance whether a statment or set is being parsed. This would not be the end of the world, but makes life more difficult. Futher, the current syntax alows an empty property list, so [ a :Deciduous, :Pine ]. is valid N3. This means that { :x } is a valid statment (with no triples), which would overlap with set syntax.

Disjoint sets?

There is an issue as to whether {$ :a , :b, :c $} imlies that a, b and c are distinct. There was a discussion of this in the SWIG.

If sets are disjoint:

• You can say how many members are in a set.
• You cannot form the union or intersection of two sets unless all the members involved are known to be disjoint, (or one knows whcih ones are equivalent), for example if one knows that they each are members of a larger set.
• In applications where the assumption is that a set is disjoint, the system can check and trap an errro if two members turn out to be the same.
• Cwm in smush mode, --mode=e, when it takes into account equality, would probably remove dupliactes from sets but there would be no signifince.

If sets are not disjoint:

• You don't know how many members they have, in general.
• You can do set union, but not intersection.
• You can validly handle sets where you don't actually know how mny distinct (people say) there are.
• Cwm in smush mode, --mode=e, when it takes into account equality, would on loading a new equality, in some cases reduce the number of members of a set mentioned in the knowledge base.

One possibility is to build into the processor that in a mode in whcih it is aware of equality it also tracks disjointness, for example using inverse functional properties and functional properties with numeric ranges.

Of course, where all the members of a set are vlues of a datatype which provides a binary equality operator, such as integers, this is not a problem.

Considered design alternatives in other areas

(older)

1. Using : for >- and -> so that the propertylist looks like a list of attributes. Advantages: really human readable. Disadvantage: keep "=" as an operator. Also, I don't like "=" being used for something which is not equality. It is ingrained as a binary reflexive operator and it would be confusing to use it in attribute attribution. Alternative alternative: use ":" for both "->" and "-<".
2. Use/allow keyword "has" for >- and "is" for <-. Maybe, if still unambiguous, allow "of" for both "->" and "-<". And/or use colon instead of "of". These assume that the english words people pick as properties are noun clauses. I actually preferred the use of verb clauses for what is in fact a verb. I used to prefer "wrttenBy" to "author". Now I have found the role-noun form much better.
3. Making the subject of the propertylist, be another property. (Say, "ref"). This is like Henrik's SOAP-RDF mapping. Every statement has to become an anonymous node syntax example: [ >- core:ref -> [ >- x:firstname -> "Ora" ] ; >- dc:wrote -> [ >- dc:title -> "Moby Dick" ]]. The thing becomes a binary rather than ternary syntax so we should use binary syntax. Using -< and -> only (omitting the >- and <- ) example would be

   [ core:ref : [ x:firstname : "Ora" ] ;

   dc:wrote : [ dc:title : "Moby Dick" ]

   ]

   or equally well

   [ x:firstname : "Ora" ;

   dc:wrote : [ dc:title : "Moby Dick" ]

   ]

   We need better examples, requiring explicit reference to the subject by URI.

4. Allowing well-formed XML element as object. reserve <alpha for this? What does XML infoset look like expressed in RDF in notation3? decide: don't do it. Burdens notation3 compiler with XML parser weight.
5. Use <> for URIs instead of ' - DanC. Hmmmm I wanted to keep <> for other things maybe like string delimiters. Actually it is cool to use inverse <. for stings >this is a string< because then you end up being able to make pages which look like markup and which are functions in notation3.
6. Bind vs @prefix. Bind was a directive which declared a namespace with an implicit "#" between the namespace and the local name. This has many advantages: it meant that by looking at a URIref one could separate it unambiguously into namespace URI and fragment ID. This in turn meant one could dereference the namespace URI to get a schema or other information describing the namespace. However, this is not standard RDF. Nevertheless, the use of namespaces ending in "#" is recommended, as then the items in the name space can be easily described by a single document associated with the namespace identifier.
7. Whitespace:  what about unicode NL? This was included as one  of teh few changes which happened in XML as it changed fro 1.0 to 1.1 . NL is a C1 control character which was introduced to allow the EBCDIC newline character to eb encoded.  Why should one have a separate NL from the LF which CCITT defined all those years ago as the code to be used when newline (CR LF together) was required?

Fodder

Connolly points out: "This grammar starts to look a lot like the formalized english/conceptual grammar stuff. > http://meganesia.int.gu.edu.au/~phmartin/WebKB/doc/grammars/ > http://www8.org/w8-papers/3b-web-doc/embedding/embedding.html

Philippe Martin says, "Given the similarities of your Notation 3 with the (currently) more readable and expressive Frame-CG notation (FCG) that I designed 2 years ago and that is one of the notations used in my large-scale knowledge server WebKB-2 , you might want to have a look at some executable example files (e.g. ) and at the grammar. The wide range of "quantifiers" is especially useful. You are welcome to copy any part of the FCG grammar into your Notation 3. (email 2001/09/17)

Footnote

Thought process behind implicit definition

How does one label a node in notation 3 for incomming reference? (The quivalent of "rdf:id=")? How about a property "Thought process behind implicit definition How does one label a node in notation 3 for incomming reference? (The quivalent of "rdf:id=")? How about a property "is hereby defined to be" with a suitable shorthand? One can then refer to such as thing internally as '#foo' which is a bit messy but not bad. You can't have keywords and identifiers both using that precious status of pure alphanumerics unless you reserve keywords. [ >- n:def -> '#ora' ; >- x:firstname -> "Ora" ] . [ '#ora' >- dc:wrote-> [ >- dc:title -> "Moby Dick" ] ] . [ >- x:firstname -> "Laura" ] <- x:hasChild-< '#ora' . or equally well [ >- n:def -> '#ora' ; >- x:firstname -> "Ora" ] . [ '#ora' >- dc:wrote-> [ >- dc:title -> "Moby Dick" ] ] . [ >- x:firstname -> "Laura" ] <- x:hasChild-< '#ora' . Ah. Now consider what is the difference betwen reference and definition? I conclude there is none, as both are the assertion that the resource in question is identified by a URI. In the statements: [ >- n:def -> '#ora' ; >- x:firstname -> "Ora" ] . [ '#ora' >- x:lastname -> "Lassila" ] . is there any significance that the node '#ora' is defined to be one which has firstname "ora" and lastname "Lassila" whichever way one looks at it. I would therefore propose that the use of a new local symbol :foo or '#foo' is taken as introducing it, but the definition of it by the document is really the whole web of statements which involve it. In fact, it maybe rather difficult to talk about the definition of it as distinct from the document, as as it is always best to avoid extra concepts, I won't. The above examples should just be, therefore, [ '#ora' >- x:firstname -> "Ora" ] . [ '#ora' >- x:lastname -> "Lassila" ] isn't that simpler?.is hereby defined to be" with a suitable shorthand?

One can then refer to such as thing internally as '#foo' which is a bit messy but not bad. You can't have keywords and identifiers both using that precious status of pure alphanumerics unless you reserve keywords.

[ >- n:def -> '#ora' ; >- x:firstname -> "Ora" ] .

[ '#ora' >- dc:wrote-> [ >- dc:title -> "Moby Dick" ] ] .

[ >- x:firstname -> "Laura" ] <- x:hasChild-< '#ora' .

or equally well

[ >- n:def -> '#ora' ; >- x:firstname -> "Ora" ] .

[ '#ora' >- dc:wrote-> [ >- dc:title -> "Moby Dick" ] ] .

[ >- x:firstname -> "Laura" ] <- x:hasChild-< '#ora' .

Ah. Now consider what is the difference betwen reference and definition? I conclude there is none, as both are the assertion that the resource in question is identified by a URI. In the statements:

[ >- n:def -> '#ora' ; >- x:firstname -> "Ora" ] .

[ '#ora' >- x:lastname -> "Lassila" ] .

is there any significance that the node '#ora' is defined to be one which has firstname "ora" and lastname "Lassila" whichever way one looks at it. I would therefore propose that the use of a new local symbol :foo or '#foo' is taken as introducing it, but the definition of it by the document is really the whole web of statements which involve it. In fact, it maybe rather difficult to talk about the definition of it as distinct from the document, as as it is always best to avoid extra concepts, I won't.

The above examples should just be, therefore,

[ '#ora' >- x:firstname -> "Ora" ] .

[ '#ora' >- x:lastname -> "Lassila" ]

isn't that simpler?.

The discussion above about the universality of URIs (Universal Resource Identifiers) mentions briefly how URIs are designed to encompass both things we think of as addresses and those we think of as names. Much of the discussion of this issue has been clouded by attempts to distinguish names from addresses. The term "identifier" was picked in an attempt to side-step this issue but historically, that did not prevent a quagmire of circular discussion which in some circles paralyzed any forward progress. Therefore, in this section let me state the philosophy which to my mind sets this problem in the right light and should prevent further fruitless discussion.

There is the commonly held belief that names and addresses are different and distinct. We learn the importance of the difference between identifiers in a programming language and addresses within a computer memory. We learn the difference in properties between fully qualified domain names on the internet and internet protocol addresses. This can lead us easily into imagining that there are two types of objects: Names, which once attached to an object follow it for its life wherever it should reside, and "addresses" which change frequently whenever an object moves or is copied or replicated from one "location" to another.

However, the only true location is a point in three dimensional space, and within computer systems and especially networked computer systems there is a very large number of complex indirection between almost anything we would call a name or an address and the actual physical location of the memory cell which stores it. At one end of the spectrum a computer memory address often is really an address within a virtual memory space allocated to a particular project, and when used is translated by the hardware into a physical memory address, or for that matter into an address, into a piece of memory which is being moved out into somewhere and swapping the file on disk storage. Filenames are mapped though mount tables and directory files into "inodes" which are mapped onto track and sector locations. Internet protocol addresses [IP Addresses] similarly are not bound absolutely to a given computer: they can be re-allocated within the constraints that because they are used for routing, there is information connecting parts of the IP address with routing information and so the computer corresponding to a given IP address cannot be moved far in the routing structure. So, we see that the constraint on how you can re-use an address is a function of what information is in the address. When most programs or people mention IP addresses, they simply quote four decimal numbers, each between naught and 255 without worrying about the internal structure. So, the information within the IP address which prevents it being re-used in a different area is to most people not explicit: It is, if you like, hidden within there as the reason why IP addresses can't be used. When we want to use something to refer to a computer but still be able to move the computer or at least the thing corresponding to that identification across from one part of the internet to another, we use our domain name. The domain name system, being completely independent of the routing system, allows us to allocate any IP address at all to a computer of a given domain name. Therefore, if we believe the naming myth the domain name is a name and the IP address is truly an address.

Two anecdotes about names and addresses

Two real-life anecdotes illustrate the dangers of making this assumption. When there were only a few web servers and I kept a registry of all those which I knew, I was contacted by a group in Australia who were putting up a server with some interesting botanical information. They sent me some details of the server to be put into the list and they gave me the IP address of the machine. My email reply explained that I always prefer to refer to servers by their domain name rather than their IP address and asked them for the domain name of the server. They replied that the domain name they would use would depend on the department within the university which was responsible for maintaining the server but due to a university re-organization, it was not at this point clear which department that would be. However, they explained that they could guarantee that the IP address of the server would remain unchanged for a long time.

Several years later, the list of servers now abandoned as a single list of all World Wide Web servers was among the now-extensive web of information maintained on the server known as info.cern.ch, the first World Wide Web server set up at the start of the World Wide Web project. At this time the responsibility for the coordination of World Wide Web protocols was shifting from CERN to MIT/LCS and the embryonic World Wide Web Consortium. For a while, CERN continued to maintain the server, but later the master sources for that information were maintained in America. Soon after this the authorities at CERN requested that the name info.cern.ch should no longer be used to refer to this information, as it was no longer under control of CERN and they could no longer assume responsibility for it. In fact, there was a policy that names in the cern.ch domain should never be allowed to refer to Internet addresses which were not physically on the CERN site. Therefore all hypertext pointers into the info.cern.ch space have had to be changed over the course of time to point to the w3.org space.

These two examples show the "name" of objects having to be changed even though the objects retained their essential identity. The reason was in each case imbedded information in the name: the domain name on the server contains authority information about the maintainer of the computer whose address corresponds to the domain name. If the authority for an object changes, whether it "moves" on not, then there may be a need to change its name under these circumstances. It turns out that for almost any naming or addressing system in which there is some information (other than random numbers or dates of creation of the objects) built into the name that the name might have to be changed when the facts corresponding to that information change. Therefore it becomes simply a matter of choice between naming or addressing systems as to what sort of information you wish to include implicitly or explicitly within your "name" or "address".

Why Names Change

See also:

• In the Syyle Guide for Online Hypertext, Cool URLs don't change

It is worth looking at some of the reasons for names in practical use to change or need to be changed. Some World Wide Web servers have unwisely simply mapped the URL space onto a Unix filename space, and the results of this, especially in the early days, were URLs which might look like this:

http://pegasus.cs.foo.edu/disk1/students/romeo/cool/latest/readthis.html

Looking at the segments of this name we can see as many reasons for the name to need to be changed.

The "http:" will only be changed if the document is later served up using a different protocol and, in fact, that is probably one of the least likely pieces to change.

"Pegasus", the name of the computer, probably has a significance within the university as a computer dedicated to some particular tasks such as supporting personal student activities, and maybe maintained by a particular department or may even be a name from a project for which the computer was originally put into use before it became shared with general user space. So, "pegasus" will be changed whenever the function of supporting this particular student's web pages has to be shared with other functions.

"Cs" indicates the computer science department, so the document is bound to the computer science department. It may not be something which the computer science department has a lot of interest in, and the student may well transfer his or her interests to other departments in the future.

The name of the university "foo.edu" will probably last for a good while, though whether the university wants to continue to be associated with the document for more than two or three years is questionable.

The next section of the path, "disk1", is clearly a mistake. In fact, of course, disc1 is just a name which can be attached to any physical disk, but by grouping together all the students on a certain disk in this arbitrary way, one makes a binding between all the documents which they create which will have to be broken whenever the computer is reorganized. In fact, the relocation tables which most servers support allow much translation of names to take place and make this sort of path quite unnecessary.

The next element identifies Romeo as a student which may change even though he continues to study for the rest of his life, and then the next path element "romeo" identifies the author of the document. As in the case with CERN above, the original author of a document may later not wish to keep maintenance or responsibility for ongoing versions. For example, the document may be submitted to an organization which publishes it and formally takes over responsibility for its upkeep; it may achieve a status of some kind as a standard or an accepted thesis which causes its maintainers to change. The original author may in fact deliberately simply pass on authorship of the document to someone else. In any of these cases the name would have to change, and all references to that name would break.

The student himself has not been very wise with his choice of path name. For many people, what is "cool" changes with time and for most people what is "latest" changes with time.

Perhaps the unlikely to change piece of information in the URL "readthis" as it contains no information at all, just like the proverbial "click here". Effectively, it is a random name assigned to the document and as such, is perhaps the safest part of the path.

The last element of the path, "html" is not strictly necessary with most servers, as at least some servers will, given a URL of  "readthis" ,  serve up the data from a file which is called "readthis.html". Here the student is making it difficult for himself later to change the format or formats in which the file is available, without at least some confusion. Suppose, for example, that he later decides that the information is worth providing in audio format for blind readers. The CERN server can easily be configured so that clients specifically requesting audio formats in preference to HTML can be served as preferentially whereas more normal clients will get the HTML. So, here again is a part of the path which may be later regretted.

You can play this game with almost any name and address in any system, and it is interesting to ask yourself in each case: to what extent do I call this a "name" and to what extend do I call it an "address"? So, in conclusion we see that any information explicitly owned or implicitly included in a name is a threat to its longevity.  We see  that the difference between a "name" and an "address" is not so fundamental.  That is why

What's in a name?

Why is information included then? Generally, the information is included because in order to discover anything about the name, one has to "dereference" the name. Typically this uses some official or unofficial set of indexes distributed or otherwise to look up the name. Many names are hierarchical in the authority which allocates them. DNS names are a good example. Road names within towns are another good example. Therefore to find out where the new "North Street" is located in small town one goes to the town for the definitive answer. For information as to where the server "pegasus.cs.foo.edu" is, one must send a message directly or indirectly to a server controlled by the Foo University.

Is it possible to omit all such information from a name? Certainly. Message identifiers in mail have only the need to be unique. So, whereas hierarchical names and time stamps may be used to help make such identifiers unique, you cannot dereference the names at all. Perhaps we should call these "identifiers" rather than "names". Within a certain context, it is extremely useful to be able to refer to a mail message by its mail identifier. We say that these identifiers support the notion of equality: even though they cannot be dereferenced, you can test two mail messages to find out whether they are in fact the same simply by testing their identifiers. You can also within a finite set of mail messages look up a message of a given identifier. You just can't do this on a global scale. So this then is the essence of the naming problem:

Naming: A social and contracual Issue

Many, many solutions to the naming problem have been attempted and successfully deployed in different circumstances. At one end of the scale, it would be in fact possible using a huge network of hash tables around the world, to keep a hash index of all randomly generated unique names. The problem with this idea is that there would have to be one single funding model and one homogeneous quality of service for all names. There would be no way to pay more for a more persistent name.

At the other end of the scale, hierarchical systems such as the domain name system, and the x500 name system, have been implemented. Suppose one wants to use a name which can be dereferenced and therefore must put some information in it. That information will lead us to some authority or some root to dereferencing the name. How can we maintain the lifetime of that name as something which can be dereferenced? The only way is that we have a contract with all the agencies which are involved in supporting the systems which dereference that name that they should continue their operation giving a certain quality of service for a certain period of time.

Suppose the Foo Alumni Association ran a URL service in which a special name such as "http://alumni.foo.edu/1998/romeo/202-aab" would be available to any graduating paying their dues, and maintained indefinitely (perpetual care) on receipt of a suitable endowment.

Of course, as organizations disolve and mutate, there is nothing to stop one organization from taking over the support of  the archives another.  Forthis purpose, it would be very useful to have a syntax for putting a date into a domain name.  This would allow a system to find an archive server.  Imaging that, failing to find "info.cern.ch", one could search back and find an entry "info.cern.ch.1994" which pointed to www.w3.org as a current server holding archive information for info.cern.ch as it was in 1994, with, of course,  pointers to newer versions of the documents.

Quality of Service

Looking at an "http:" URL, while some look more sensible than others, it is not immediately evident whether great pains are being taken to make the name very persistent.  We have just discussed such a range of reasons why names can change, and clearly the social and contractual arrangements can be quite involved, so it is clearly difficult to simply define a quality of service for naming.  However, defining some well known quality of service levels would be a very useful task. This is the sort of task ideally suited to a group of trechnologies, librraians or archivists.

 In any event, for identifiers in the http space and many others, it would be useful to be able to assert what the quality of service is. This is information about a URI and a resource.  Like the information about generic URIs, it is about the sort of identity between the URI and the resource.

In his book Goedel, Escher, Bach, the computer scientist Douglas Hofstadter ruminates on self-referential systems. At times, he uses the approach of a Socratic dialogue between two characters from Xeno's fable, Achilles and the Tortoise. The conclusion of several hundred pages of musings around Bach's fugues, Escher's recusive drawings, and Goedel's theorem are that you can't try to distinuish wishes from metawishes, or the whole system breaks down. Without drawing too many parallels with the recent XML-URI discusssions, we would like to relate a conversaion between Achilles and the famous tortoise, recently overheard in a library.

[Achilles and the Tortoise are each strolling in the library. They meet.]

Achilles: Ah, Mr. Tortoise, I thought I might find you in the library

T: And a very nice library it is too, Achilles.

A: Thank you. It was a communal effort. As were the books. There are so many really beautiful books in the library.

T: And now we have dictionaries!

A: Yes, dictionaries are very important to me, Mr.. Tortoise. I want to use them to understand what some of those books mean.

T: Let's not discuss meaning, please Achilles -- you know what happens when we do that! I want to use these dictionaries in order to check that the books are correct.

A: Well, at least we are agreed that dictionaries are a good idea.

[they round a corner]

T: Achilles, what is that?!

A: Why, a dictionary, Mr. T.

T: But it is in the library! I thought when we defined dictionaries we agreed it was "not a goal" to register dictionaries in the library!

A: But surely that doesn't stop me putting one in the library?

T: Irony heaped on Irony! The Library is for books. That you should abuse it so! A dictionary is not a book. It is a metabook.

A: What? Of course it is book!

T: You said that you wanted it have the form of a book so we make them out of paper -- but that doesn't mean the intent was to put it in the library!

A: But this is my section of the library -- it is the section on Library Architecture and I need a dictionary to define the terms used in that field.

T: But you know that people can loose things in a library, and libraries can burn down ... there are so many reasons that dictionaries should not be in the in the library, Achilles!

A: Look at this way, Mr. Tortoise: when I am doing research in the library, I need to be able to look up words, and so I need a dictionary in the library.

T: You have some woolly notion of finding out what books mean, Achilles, but we haven't agreed about that. The meaning of the semantics of "meaning" are not a consensus in current linguistic epistemorthosemantisophologic theory.

A: I don't need to go into that, but I need a place for dictionaries.

T: Oh, we have all been discussing where dictionaries should go. We have plenty of ideas: We have plans for a new vault building down the road much more secure than this library. We have that white tower on the hill we could use too.

T: Besides, in practice, most of us keep a pocket dictionary for each language we use in our briefcases. It isn't as though we need so many dictionaries. Frankly, dictionaries have such different requirements to books I am shocked to see this dictionary in your section of the library! If you don't take it out out, I will bite your heel.

A: But I thought when we designed the library it was so that any sort of book could go in it. That is why we called it the Global Eternal Bibliotech, after all: it is Good for Every Book. I should be able to keep this dictionary in it simply because it is a book.

T: But Achilles, for the last time, a dictionary is not a book!

Most of these notes are about architecture at the web layer. However, a healthy web for society places requirements also on the Internet layer.

In 2008, this was threatened in the UK by the company Phorm proposing to use data from deep packet inspection (DPI). The system would use special apparatus at the Internet Service Provider (ISP) to monitor traffic, peek inside the IP packet's payload, and determine every URL looked in a household's browsing on the web. This profile would be used to provide taregetted advertizing. They also planned to automatically "protect" users by redirecting any access to blacklisted (phishing, etc) sites.

A discussion was held at the House of Lords by Baroness Miller on 2009-02-11. These are some notes I made for the event, which I attended.

1. The Internet in general has and deserves the same protection as paper mail and telephone.
2. If fact you could argue that it needs it more, as it carries more or our lives and is more revealing than our phone calls or our mail.
3. The access by an ISP of information within an internet packet, other than that information used for routing, is equivalent to wirtetapping a phone or opening sealed postal mail.
4. The URLs which people use reveal a huge amount about their lives, loves, hates, and fears. This is extremely sensitive material. People use the web in crisis, when wondering whether they have STDs, or cancer, when wondering whether they are homosexual and whether to talk about it, to discuss political views which may to some may be abhorrent, and so on.
5. We use the internet to inform ourselves as voters in a democracy. We use the internet to decide what is true and what is not. We use the internet for healthcare and social interaction and so on. These things will all have a completely different light cast on then if the users know that the click will be monitored and the data will be shared with third parties.
6. The URLs produced when using forms contain the information typed into those forms. Personal data, private data.
7. If people really want privacy, then many users and sites may switch to using SSL encryption: to doing theior actual web surfing thorugh an encrypted tunnel. This takes a lot of server CPU cycles, making server farms more expensive. It would slow the user's computer. It would effectively slow down the whole net. It also prevents the use of HTTP proxies, which currently help the efficiency of web access.
8. There are considerable risks if the information is abused. Imagine:
   • To be able to buy a profile of a person you are interested in;
   • To discriminate based on profiles of people when deciding whether suitable to employ them;
   • To discriminate in giving life insurance, and so on, against those the have lookup up (say) cardiac symptoms on the web;
   • Criminal attacks on government officials at home;
   • Foreign attacks on the country made by targeting and analyzing key individuals;
   • Predators choosing, stalking, and targeting victims;...

   to name a few.

9. The information could be deliberately abused by an inside worker, or could be acquired by an attack on the system's machines.
10. The power of this information is so great that the commercial incentive for companies or individuals misuse it will be huge, so it is essential to have absolute clarity that it is illegal.
11. To put his in perspective, it is like the company having a video camera inside your house, except that it gives them actually much more information about you.

The act of reading, like the act of writing, is a pure, fundamendal, human act. It must be available without interference or spying.

Acknowledgements

Thanks to colleagues who reviewed these notes and provided useful feedback, including Hal Abelson, Karen Myers, Thomas Rössler, Amy van der Hiel, and Danny Weitzner

References

Phorm in Wikipedia http://en.wikipedia.org/wiki/Phorm

The author on BBC news disapproving of the spying on people's URLs: http://news.bbc.co.uk/2/hi/technology/7299875.stm

Notation 3 Logic

This article gives an operational semantics for Notation3 (N3) and some RDF properties for expressing logic. These properties, together with N3's extensions of RDF to include variables and nested graphs, allow N3 to be used to express rules in a web environment.  

This is an informal semantics in that should be understandable by a human being but is not a machine readable formal semantics. This document is aimed at a logician wanting to a reference by which to compare N3 Logic with other languages, and at the engineer coding an implementation of N3 Logic and who wants to check the detailed semantics.

These properties are not part of the N3 language, but are properties which allow N3 to be used to express rules, and rules which talk about the provenance of information, contents of documents on the web, and so on.  Just as OWL is expressed in RDF by defining properties, so rules, queries, differences, and so on can be expressed in RDF with the N3 extension to formulae.

The log: namespace has functions, which have built-in meaning for CWM and other software.

See also:

• The schema for the log: namespace
• A vocabulary for expressing differences between RDF graphs
• a formal design for RDF/N3 context/scopes
  Dan Connolly to www-rdf-logic, Thu, Sep 06 2001

The prefix log:  is used below as shorthand for the namespace <http://www.w3.org/2000/10/swap/log#>. See the schema for a summary.

Motivation

Formal syntax

{ @forAll <#h>. @forSome <#g>. <#g> <#loves> <#h> }.

@prefix log: <http://www.w3.org/2000/10/swap/log#>.
@keywords.
@forAll x, y, z. {x parent y. y sister z} log:implies {x aunt z}

{x parent y. y sister z}

{x aunt z}

Joe parent Alan.
Alan sister Susie.

Joe aunt Susie.

@forAll x, y, z.
{ x wrote y.
  y log:includes {z weather w}.
  x livesIn z
} log:implies {
  Boston weather y
}.

Bob livesIn  Boston.
Bob wrote  { Boston weather sunny }.
Alice livesIn Adelaide.
Alice wrote { Boston weather cold }.

Boston weather sunny.

I was recently asked to talk about the idea of "open", and I realized the term is used in at least eight different ways. The distinct interpretations are all important in different but interlocking ways. Getting them confused leads to a lot of misunderstanding, so it’s good to review them all.

When we tease apart their meanings, we can understand more clearly which aspects of each are the most important. The first, one of the most important forms of openness for the Web, is its universality.

Universality

When I designed the Web protocols, I had already seen many networked information systems fail because they made some assumptions about the users – that they were using a particular type of computer for instance – or constrained the way they worked, such as forcing them to organize their data in a particular way, or to use a particular data format. The Web had to avoid these issues. The goal was that anyone should be able to publish anything on the Web and so it had to be universal in that it was independent of all these technical constraints, as well as language, character sets, and culture.

Net Neutrality is essential to an open, fair democracy Close to the principle of universality is that of decentralization, which means that no permission is needed from a central authority to post anything on the Web, there is no central controlling node, and so no single point of failure. This has also been critical to the Web’s growth and is critical to its future.

Open Standards

The actual design of the Web involved the creation of open standards – and getting people to agree to use them globally. The World Wide Web Consortium (W3C), of which I am the Director, helps create interoperable standards for Web technology, including HTML5, mobile Web, graphics, the Semantic Web of linked data, and Web accessibility. Any company can join and anyone can review and help create the specifications for the Web.

The W3C process emphasizes transparency, openness, and consensus. In practice, fairness, technical quality and speed of process are always a trade-off to balance.

Other bodies work on other layers of the design. The IEEE for physical internet connectivity and the IETF for internet interoperability, for instance. Organizations like the IETF, IEEE and W3C support ‘OpenStand’ which encourages the development of market-driven standards that are non-national, free to access, open to participation, and (for W3C) free of royalty payments.

Open Web Platform (OWP)

W3C’s Open Web Platform is the name for a particular set of open standards which enable an exciting stage of Web computing. Standards such as HTML5, SVG, CSS, video, JavaScript, and others are advancing together so that programmes that once worked only on desktop, tablets or phones can now work from within the browser itself. It has all the power of HTML5, like easily-inserted video and, in the future, easily-inserted conferences. It also features the APIs for accessing hardware and other capabilities on the device, such as a smartphone’s accelerometer, camera, and local storage. While native apps are limited, Web Apps can work on any platform.

With Web Apps, every Web page can become a programmable computer, whether it is on a mobile device, a desktop, a TV, or, in the future, a car console. Native apps are not on the Web, they’re not part of the Web. Native apps that work on a single platform or even a single device are thus more limited than a Web App. I therefore encourage everyone to build Web Apps.

Open Government through Open Data

In 2009, I resolved to encourage more use of data on the Web. Too many websites could generate nice reports as documents, but had no way to access the data behind it to check and build on the results. In February that year I stood up in front of a TED audience and asked them for their data; I even got them to chant: “raw data now�. In April that year, I met with Gordon Brown, then Prime Minister of the UK and with him began the UK Government’s ground-breaking work on Open Data. That same year President Barack Obama announced his commitment to the US Open Government Initiative. In 2010 I went back to TED and showed the audience some of what had been achieved, including Open Street Map’s role in relief efforts in Haiti. It’s important to me that I can get at the source code of any software I’m using. In 2012 we launched in the UK’s Open Data Institute (ODI). ODI is a non-profit institute that incubates startups and promotes open-data businesses in East London’s Tech City. ODI was created to take advantage of and to help guide the wave of open data adoption by business that is happening now. It is an exciting time for open data, but there is a huge amount more to do.

Update 2024: Open Science

We have just (May 2024) celebrated the 70 years of the European Particle Physics Lab, CERN, in Geneva. A lot of the talks were about the importance of Open Science.

Openness with personal data on the Social Net

The word “open� is often used in the sense “I wouldn’t be that open with my personal life�. We are as a society learning how to draw the right boundaries in this new age. I won’t go into this in detail, but connected issues include the extent to which a social networking site which helps people share information also benefits from the data in completely different unforeseen ways, and ideas about what different sorts of data about people should be used for anyway. These issues may lead to be cultural norms as well as possibly new technical architectures.

Open Platform

While it’s not really a feature of the Web, a concern for a lot of people is whether they can choose which apps run on their own phone or computer. An Open Platform means having the right to install and write software on your computer or device. One motivation to close off a computing platform comes from a manufacturer wanting to allow you to experience their content on your machine without being able to store it or pass it on. Some systems are very closed, in that the user can only watch a movie or play a game, with no chance to copy anything or back it up. Some systems are very open, allowing users to take copies of files and run any application they like. Many systems fall in between, letting users pay for additional material or an experience.

The W3C community is currently exploring Web technology that will strike a balance between the rights of creators and the rights of consumers. In this space in particular, W3C seeks to lower the overall proprietary footprint and increase overall interoperability, currently lacking in this area.

In the US particularly, the situation is aggravated by the Digital Millennium Copyright Act (DMCA) and the Computer Fraud and Abuse Act (CFAA), two laws which allow a person who uses a computer improperly to be jailed as a felon for a long time. These unjust laws colour the debate so much in the US that some people react by saying that all platforms should be completely open, so that no one can be said to use them improperly. Hopefully these laws will be fixed through debate about how to balance the needs of creative people to be paid and the needs of consumers to be able to contribute but also to be able to rip, mix, quote and archive this material.

Open Source

“Open Source� is another way “open� is used on the web, one which has been and is very important to the Web’s growth. It’s important to me that I can get at the source code of any software I’m using. If I can get at the source code, can I modify it? Can I distribute the modified code and run it on my machine? As Free Software Foundation lead Richard Stallman puts it, “free as in freedom rather than free as in beer�.

Open Access

Open Access is a Web-based movement specifically about free (as in beer) access to the body of academic learning. Governments, and therefore taxpayers, pay for research via grants but often the results of the research are kept in closed-access academic journals. The results are only available to those at big universities. The poor and those in remote rural areas cannot participate. Open Access journals are academic journals legally and technically available openly on the Web at zero cost.

These have to be funded either by publication fees, which in turn have to be agreed to by research funders, or through the implementation of a very low cost Web-based system. Nowadays, governments (with the US NIH taking a lead), and the European Commission are starting to require open access to the results of taxpayer-funded research.

Open Internet and Net Neutrality

When we talk about keeping the internet free and open, we are often worried about blocking and spying. One of the ways in which we protect the Web is by ensuring Net Neutrality. Net Neutrality is about non-discrimination. Its principle is that if I pay to connect to the Net with a certain quality of service, and you pay to connect with that or a greater quality of service, then we can both communicate at the same level. This is important because it allows an open, fair market. It’s essential to an open, fair democracy. The alternative is a Web in which governments or large companies, or frequently a close association of the two, try to control the internet, with packets of information delivered in a way that discriminates for commercial or political reasons. Regimes of every sort spy on their citizens, deriving hugely accurate and detailed profiles of them and their intimate lives. Today, the battle is building. The rights of individual people on the Web are being attacked, and at the moment only a few few people really understand and realize what is going on.

The World Wide Web turns 25 next year. We have come a long way, but we must all continue to push for these various forms of openness in the appropriate places. Only then can we ensure that the Web is for everyone.

2024-09-28 Update: Open Process

Many the projects I've been involvd, or depended on, are developped with an open process. The W3C and the IETF are open processews where there are non-profit Organizations to run them. Many open source projects, are based on open processes aroud a shared code repository on something like GitHub or Gitlab. Hamish Campbell includes them in his essay on 4 opens[HC]:

Open process refers to the transparent and participatory decision-making processes that govern [tech] projects. By involving stakeholders in project planning, development, and governance, open process fosters trust, accountability, and collective ownership. Progressive social and tech initiatives can embrace open process by adopting democratic and inclusive decision-making structures, such as consensus-based decision-making or participatory budgeting. For example, open process can enable community-led initiatives, address social justice issues, and promote collective well-being.

He mentions tech projects, but there is no reaon it has to be a tech project. It could be any project.

Here we look at the relationship between documents (living or dead but basically bits of state) and messages (events with associated data, including typically but not essentially sender and recipient).

Here is a proposal for a project: "Paper trail" state machine for workflow. The concept here is that the state of any transaction is in the real world (and in this formalization in the Web) just a function all the messages which form part of a protocol.

Epilogue (2001/05)

The Web Services workshop, in discussing transactios over the Net, surfaced the need for process flow descriptions

Update (2004/03)

The Semantic Web Application Platform (SWAP) now has enough functionality to implement these ideas. see ppt-bank, especially checking.n3

Introduction

Social processes look like state machines. However, they don't exist as a state variable stored in one place, but as a trail of documents. You know the true state of the machine only if you have access to the latest documents. (This is not the problem addressed here, this is real life being modelled.) Paper-trail is a system which allows one to follow a strict process by creating new documents in a constrained fashion. Every paper-trail document has a pointer to a "paper-trail schema" which defines its document type (eg "constitutional amendment") a pointer to its justification documents (maybe) a notarization of when it was checked against the schema by the paper-trail program. The schema defines:

• Prerequisites for a document being valid, in terms of other documents
• Hints to other document types you can make from this one (state transitions)

Example

To make a new W3C working draft, the schema requires pointers to old working draft new document, and editor's authorization. The editor must be defined as editor on home page of working group where working group page is pointed to be by old draft. If all those exist, then the new document is created from all that and notarized (time stamped) by the software. The human readable part of the document is created as a (simple macro) function of the input documents. A document also has a buttons to take you to a form to turn it into another type of document according to hints in the schema.

Example

A button on a Working Draft takes you to a form for promoting it to a "proposed recommendation". This requires different things (all the above plus endorsement of new draft by director or any two members of the management group.)

Technology

If you are considering this as a student project, consider these directions:

• Use RDF within the document to express its state.
• Develop declarative language for defining the prerequisites - ideally in RDF too.
• Develop GUI for creating a new document by supplying the prerequisites
• Allow hooks for digital signature but don't have to implement it

Generalizing for formal protocols

The concept of a paper trail is common in conventional administration, but the model can also be applied to well-defined computer protocols.

Model

The model is that a protocol P defines a status s_n as a function of a message m and a previous state s_n-1, and the time t.

s_n= P(m_n, s_n-1, t)

or for that matter as a function of all the messages to date

s_n= P'({m_i}_i=1..n)

The state could be a logical formula, an RDF graph, or an XML document, or just a number, in decreasing order of interest. The system can be a any one of a number of types of machine, including the well-known finite state machine and push-down automata.

In an XML world, think of the state and the messages all being expressed in XML, and the protocol maybe being an XSLT script.

The state must record everything necessary for calculating future states for any new message. It could also record the results of the protocol. For example, the state of TCP (where IP packets are the {m} ) must hold the state of the packets unacknowledged in the sliding window, but when the connection has been successfully closed it could hold either just "terminal state", or also the ordered set of bytes transferred in the connection.

The protocol function can be seen as an information destroying function. By specifying what needs to be remembered, it defines what can be thrown away. This is of course very important. Of course, one might in some cases still want to spool the messages for security, but the actual information needed to describe the state of affairs is limited..

Typically, to be valid, messages will link back to previous messages either directly or though common threading identifiers of some sort. A message without such a reference will in most cases not have any effect on the state.

There will in general be error states, which the protocol does not allow, which any message which is invalid in some way will lead to. Functionally there need only be one error state but in practice one might want t preserve the state before the error and details of the error. Some protocols model most errors themselves by sending.

There must obviously be a set M₀ of valid ways to start a protocol in the first case from the generic initial state s₀. For example, in TCP one sends a SYN message; on the telephone one picks up the receiver. For any m in M₀, P(m, s₀) will be a valid (non-error) state.

There will in some systems be a set of F final states, in which no further messages can have any effect on the state. For any s in F, P(m,s) = s for all m.

For example, in the US, when 7 years have passed since a transaction occurred, then all records may be discarded as no one even the tax man has the right to query them. The state is reduced to a minimum. Most systems can be modelled in a simple of complex way, the simple way ignoring a lot of the auditing processes for example. A simple model of a loan between two people has a state which is the balance amount and one final state when that is zero. Other systems are designed to remain in non-final state: a lifetime warranty is a protocol which remains in non-final state (until you die!), waiting for any message that you are dissatisfied with the product.

Real system are part of bigger systems, and so the real protocol will function as part of a larger protocol. For example, a working group at W3C goes though many internal state changes, and (on a simple model) the last is when their work is accepted by the Consortium as a whole as a Recommendation. This is a message leaving the system, which forms part of the larger protocol. Modeling this is clearly interesting. (To demonstrate this nesting by an example of it breaking, think of the case of a working group not arriving at consensus and passing on not only a final document but also a minority report, basically a peek into the internal workings of the group which did not in fact arrive in its final state. ) This would include modelling tasks which can split, and be recursively delegated, and so on.

Cool things

This system can allow well-defined social processes to work eg on a net newsgroup, or by email. ie, it works in a write-only medium.

It models real life in commerce well, where the state really is an abstract thing and one's perception of it depends on the set of messages one has had access to.

Hopefully we can use this model to define systems which are even more powerfully distributed than any we use at the moment.

Linking Remote operations and Data Formats

I must have discussed the relationships between remote operations and data formats before. Maybe I have made a table with schema languages compared against interface definition languages, and so on.

Now we have a clear way of expressing the relationship between the two. A Protocol definition document defines a document as a function of messages, which can be represented as documents - so we can look at remote operations in terms of documents. Typically RPC messages are very constrained: this model allows much more complicated multi-party protocols to be defined.

Challenges if you finish early

If making a paper trail machine was fun, here are some more ideas.

• Add time-aware social processes such as promises and timeouts.
• Do you need to be able to prove non-existence of documents?
• Locally to an author or globally?
• States can split. (draft can go to W3C or IETF process or both).
• How can you limit this, when socially undesirable?)
• Develop proofs that processes will achieve given ends.
• Model processes near you:
  • auction
  • peer review journal
  • presidential impeachment ;-)
  • internet newsgroup creation
  • formation of a company
  • MIT purchasing (possible PhD thesis ;-)
• Develop theories in which players are
  • collaborative
  • competitive
  • allowed to create new schemas to achieve their ends
• Model existing systems near you:
  • TCP
  • HTTP...
• Develop a protocol machine, which, acting on behalf of one agent, will determine when that agent has a possible move to make, and when in fact the protocol is acting for that agent. Develop a GUI which helps a human user chose from the set of possible options at that state of the protocol.

Products

The thing which would come out of this idea would I imagine be a standard language for writing protocols. Of course, it would mainly be something else, such as an rdf-logic language, or prolog or whatever, but there would have to be hooks to define it to be a definition of a protocol.

This takes the self-describing web concept into a new area: that messages are self-describing in that they contain a pointer to the language in which they are written, and that includes (or points to) the protocol to which they claim to adhere.

@@ Add pointers to work done with Notation3

This is a proposal to address the problems with the persistence of HTTP URIs. It introduces the concept of a datestamped domain name with associated rights and obligations of ownership.

Introduction

 The problems of the lack of persistence of URIs lead to many well-known problems, including

• user frustration and social dysfunction with Error 404 messages; and
• (worse) the dereferencing of a URI using HTTP/DNS leading to completely different resource to that intended by the referring person.

 A second-order symptom of the problem is that there is a constant stream of proposals for new URI schemes with different, incompatible, name-lookup technology. These are often made with less attention to the real social issues surrounding persistence than HTTP/DNS, but propose to be systems otherwise similar except in having greater persistence.

The analysis

The persistence of HTTP URIs can be factored into two issues:

1. The persistence of the opaque string which follows the domain name, and
2. the persistence of the domain name  itself.

The first of these is an issues is of course under the control of the domain owner.  This, combined with a dearth of tools which help one run a web server with persistent URIs, has led to a vastly varying level of persistence.  Some sites understand and construct their URIs carefully, while some obviously have not thought about the problem and end up changing URIs out of thoughtlessness rather than malice. I have summarized some aspects of this problem in "Cool URIs Don't Change" .  This relies on publishers making an institutional commitment to persistence.  I have tried to lead the way with W3C's draft Persistence Policy. However, this only addresses the first element.

The second issue is summarized by ICANN chair Ester Dyson's line, You don't buy domain names, you rent them. This single meme instantly undermines any public expectation that domain names should be persistent. In principle, if  http://www.example.com/downloads today points to example.com's download page, tomorrow, it could point to anything as defined by a company which has bought, acquired though law suit or accidental expiry the domain name "example.com".   In fact, however, huge numbers of links are being made with HTTP. The planet's investment in domain names in references is huge. The technology which actually is involved in dereferencing such identifiers has become more and more sophisticated.  In practice, also, the legal weight  behind a significant organization's ownership of a domain name is considerable.  No one would dream that a legal battle microsoft.com or ietf.org would be lost to some sneaky entrepreneur.  Through the rift between trademark law and domain names is a problem, there in fact is strong legal support for an organization's ability to keep its name. But is this enough?

The Solution

I think we can do better. We can do better, if this scheme works out, on both issues.  To tackle the second, we can create domain names which are allocated once and once only, which are bought, not rented. We can simultaneously set expectations that such data will endure, that names will not be reused, and that information will be available even after organizations involved have disappeared.  The trick is the same one as used in W3C's datespace URIs (such as http://www.w3.org/2000/01/sw) and British car registration plates: to date code them. For example, let us create a series of top-level persistent domains y2000, y2001 and so on. One would only be able to acquire a .y2000 domain name  during the year 2000. Once acquired one would have it forever. In fact, the domain name would correspond forever to the information published in it. 

We could take the precaution of inserting a few rules for sanity here.  There should be some combination of due diligence to ensure you are not infringing trademark when applying for an alphanumeric name. I would like to put a limit on the number of persistent domains any company could own - or at least put those who have n in the queue ahead of those who have k<n.  There could be a convention that if you are happy with a random numeric domain 6872364.y2000 you can have one immediately and automatically.  It would be great to put some constraint on sitting on a domain without using it, and maybe on transfer of domain ownership.

To tackle the first issue, an organization wishing to enter the scheme makes necessarily a few commitments. One is that it must partake in some cooperating mirroring scheme in which other organizations or commercial services take on running mirrors of the site.  I can imagining this working, for example, as a for-pay service for consumers, or a mirror-ring system for academic institutes. There must be some form of contract which, in the event of the original publisher of the information coming to a voluntary or involuntary demise, the mirror sites will continue the service.  The documents then enter an archive state in which the original publisher loses authority to evolve the domain and the public gain rights of access. The actual contractual arrangements will in fact have to be skillfully set up. For example, there will be some information which will be just so uninteresting that no one will be prepared to pay for it in the long run, but there must be some way to give serious archival institutions (the major national libraries for example) the right to take over an archive for the public good.  However, it would be best to start with simple conditions, but allow them to be modified with experience (real and imagined - gedanken) of the system.  Other interesting things which come to mind include the mirroring of confidential access controlled documents with a 30 year timeout on the confidentiality.

I don't know how best to enforce that a URI is never reallocated to something else by a publisher. Actually, I don't think it will be a long term problem, as the tools will be made such that it is not a function. ("Rename: command not known"!).  Obviously re-use would cause a big mess, as caches across the globe would run out of sync, and assumptions made by mirrors and proxies would become invalid.  Perhaps a suitable disposition for a domain whose publisher consistently flaunts the rules would be for it to be more or less automatically declared dead, and for it to pass into archive state just as though the organization had passed away. The organization can then ask for a new domain and start again - at least the first time.

A clarification of what re-using a URI means.  As I have pointed out many times, most URIs are Generic URIs. They refer to, not a fixed set of bits, but a conceptual resource whose representation may vary with (for example) time, technology, language, and so on.  This is fine, even in a persistent domain. It is in my opinion important to distinguish (and ideally independently identify with separate URIs) generic resources and any specific representation in use in a specific case.  The contractual obligations of ownership of a persistent domain could usefully include the provision of such metadata, and the allocation of a persistent URI to the actual (mime-type, octet-string) specific entity.

Obligations of persistence would have to apply across the transfer of the persistent domain. The persistent domain would be property, with (like land) rights and obligations. Transfer of the domain name would carry within it both.  Fleet Bank would not be able to buy BankBoston and simply drop support for documents bankboston.y2000 - without all public documents falling into archive state.

Another clarification.  A document in archive state has a certain general license to mirror and reproduce in unmutilated form, but the intellectual property rights remain as ever with the author. A very tempting rule (which I am not convinced of yet) is that once the original domain owner has defaulted on publishing the material, that it acquires some limited redistribution license^*.  This is the Web equivalent of an implicit right to photocopy any book out of print: the publisher has deigned not to make copies, and it impedes the operation of society to prevent this erstwhile public information from being accessible.

Summary 

Here is a collection of simplified rules which would form the protocol of persistent domains.

1. Domain names *.y2000 only allocated in 2000, and so on;
2. Some level of trademark due diligence before registration for alpha numeric names
3. Random numeric names   123678.y2000 issued immediately but not transferable
4. A limit on the number of persistent domain per organization would be useful
5. Domain names owned for life and persist forever
6. URIs may be live or frozen but not reused arbitrarily
7. Implicit irrevocable license for 3rd parties to mirror public info now and after death
8. The registry(-ar - whatever is the controlling authority, I forget the terminology) would be a neutral non-profit cooperative.
9. ICANN's delegation of .y[1-9]* would be irrevocable in all time.
10. Anyone using it would pay me $1 (just kidding! :-)

Provided these are put together with sufficient care, the system should run itself in such a way as to preserve our information world for posterity, be it represented by a consumer in a decade searching for instructions from the now-defunct manufacturer of an appliance, or by a historian in a millennium trying to figure out what on earth made us all tick way back then.

Read whole article...

Harry Halpin: How did the idea of philosophical engineering come about?

Tim Berners-Lee: The phrase came about when we were originally discussing the idea of Web Science, and I was tickled by the fact that when you study and take exams in physics at Oxford, formally the subject is actually not physics but experimental philosophy. I thought that was quite an interesting way of thinking about physics, a kind of philosophy that one does by “dropping things and seeing if they continue to drop� – in other words, “thinking about the stuff you do by dropping things.� Then it came up again when trying to explain to people that when we design Web protocols, we actually get a chance to define and create the way a new world works. It struck me what we ended up calling “Web Science� could have been called “philosophical engineering,� because effectively when you create a protocol you get the right to “play God� and define what words mean. You can define a philosophy, to define a new world. So when people use your system, when they run the protocol, to a certain extent they have to leave their previous philosophy at the door and they have to join in and agree they will work with your system. So you can build systems, worlds, which have different properties. That's exciting, and a source of responsibility as well.

Harry Halpin: Would you consider the creation of Web standards to be an act of philosophy in progress?

Tim Berners-Lee: Certainly, when people write a specification, they argue about what words mean until everyone assumes that they mean in some sense the same thing. When the concepts in different people's brains have been sufficiently well-aligned and there have been enough connections between the concepts, this is written down in a language that people feel comfortable with and that they share. You can if you want to philosophically argue that a word is in fact ambiguous, but nobody bothers. Understand when you play the game [of specification-writing], you're not going to argue about that. For example, you're not going pay a bill online, and then afterwards come back and say “Well, I sent some HTTP headers off, but because their just HTTP headers, they don't actually mean anything.� As a spammer said once, “It's just a form field, I can put whatever I like there, it doesn't have to be the person sending the email.� But it does if your playing the game! I think one of the things we're missing is the relationship between the law of the land and protocols. It should be easier to establish that when someone disobeys a protocol that they've broken the law via a straight-forward path.

Harry Halpin: One of the most important aspects of natural language is that it's composed of words. In contrast, the Web is a space of URIs. How is it that URIs and their meaning differs from other possible systems like natural language? What is special about URIs?

Tim Berners-Lee: There are many URI schemes, but one thing that is nifty about HTTP URIs is that they have domain names in them. So they're hierarchical, and a domain is something that one can own. In the way the protocol works, the owner of the domain has the right to say, and the obligation to say on the Semantic Web, what the things in that domain mean. It's not a question of philosophical discussions between third parties. If there's a dispute about what a URI stands for, then the way the protocol works is that you go to the person who owns the domain name, who typically delegates it to someone else, who has in turn designed an ontology that they store on a Web server. The great thing about the Web is that you can look up the HTTP URI in real-time to get some machine-readable information about what it means straightaway.

Alexandre Monnin: Regarding names and URIs, a URI is not precisely a philosophical concept, it's an artifiact. So you can own a URI while you cannot own a philosophical name. The difference is entirely in this respect.

Tim Berners-Lee: For your definition of a philosophical name, you cannot own it. Maybe in your world, in your philosophy, you don't deal with names that are owned, but in the world we're talking about, names are owned. Some people have a philosophy where they they find it useful to think of a name as just a function of use, not of definition. Other people work in worlds like lawyers, where the model is that there is a definition, a legal definition of a term. There's enough law to insist that while meaning is use, but use is according to definition, because otherwise people could get put in jail. So there are models, and now we're adding another one, in which meaning is defined by the owner of a name.

Harry Halpin: Wasn't it controversial that when the Web was first starting that everything could be named with a URI?

Tim Berners-Lee: At the IETF certainly there was resistance. I originally called these things “Universal Document Identifiers� (UDIs) even before we started using them for concepts. The IETF were a bit put off, thinking it was too much hubris to call them “universal� but now I realize that I should have held firm and said “but they are,� as any alternative system of naming you can make out there, I can map it to the character set we use in URIs and I can invent a new scheme for it. So because we can map any scheme, we'd already mapped Gopher and FTP and these things. Now we've got HTTP and there will be lots of other schemes. So in a sense it is universal, we're saying anything, any name that you come across, can be mapped into this space. So yes, there was a lot of pushback against that, and hence the “uniform� rather than “universal� in URIs.

Alexandre Monnin: Given the origins of philosophical engineering and Web Science, don't you think that Web Science is doing two things? The Web is an artifact, we produce it, we implement it, and as you said, we decide what the protocol means and how it should be used. On the other other hand, Web Science is a science, so we make discoveries and we are also surprised by our own creation.

Tim Berners-Lee: The Web Science cycle starts off with idea that the design of the Web is not just the design of one thing but the design of two things; for example in email, there's a general technological protocol like SMTP and there's a social protocol. In email, there's a social protocol that states that everyone involved is ready to run a machine that has the space to store all your friends emails while they are en route to its destination, that people will send email to each other on perfectly reasonable topics, and that people will read email that they receive. There's a social piece of email, then e-mail is actually pushed around with SMTP and pulled off with IMAP, and those pieces then together form a system. It's a microscopic system that explains how one person sends another person an email through a finite number of hops, but then you get the effects of scale. So the engineering of Web Science is not like building a mousetrap. You design a microscopic system, but what you're interested in is the macroscopic phenomena that emerge. So when you do the science, the analysis, and the whole rest of the cycle for e-mail, you look at what is happening and notice “Spam has happened, oh dear!� What went wrong? One of our social assumptions was wrong, namely that everybody is friendly and will only send email to another person when the other person wants to read it. So the academic assumption is broken, and we have to redesign e-mail, and interestingly no-one has really succeeded in redesigning either the social or technical piece of mail to make spam go away. So there's an example: there's a design piece and an analysis piece, there's an engineering piece and a science piece, one being done on the microscopic system and the other being done on the microscopic system, and we're missing a lot of the mathematics that let us understand the connection between the two levels.

Harry Halpin: What is the role of philosophy in Web Science? Is there such a thing as a philoosophy of the Web?

Tim Berners-Lee: An awful lot of philosophy in the past has been in a wasted as it was done before we understood evolution. We were trying to understand emotions, and now we can point to evolution producing mammals with emotions. A lot of philosophy in the past is inapplicable. A lot people might say that philosophy is irrelevant to daily life, but if a Working Group stops and people start arguing about what things really mean, and people refuse to play the game, refuse to say what terms mean, and they don't do their job to define a protocol properly, then its a philosophical task to point out to them that this is important. Also, philosophy may be necessary to explain when the legal system hits the Web. When you make a web-page you can link to anything, you can write anything about it, but when a lawyer comes along and reserves the right to charge you to link to their page, then in a way it's a philosophical question, because you have to tie it to the way the protocol is defined over a name, just a reference, something that has never been controlled over the millennia. Systems where you control names haven't worked so far, and so you need the philosophy to show how these protocols ground out in legal history, in concepts for using names lawyers understand.

Alexandre Monnin: What do you expect from philosophy of the Web?

What I would like for philosophers to do is to work diligently and to produce very nice, simple, documents which describe to people like computer scientists how things work in a simple way. What happens when you click on a link? Quite a lot of that is philosophy. So, I'd for you to have enough of a body of understanding, so when people in a Working Group stop and say “Wait, this doesn't match what I learned from Wittgenstein� that you can say “No, please go read this pamphlet, its about philosophical engineering and it explains the philosophy of what your doing, so you won't find Wittgenstein very useful in this case or these are the bits that you will find useful.� So if you can produce enough discussion and understanding so that we don't have to stop work for philosophical discussions and we can rely on it being there, that would be excellent.

Read whole article...

Preface

Architectural and philosophical points

These statements of architectural principle explain the thinking behind the specifications. These are personal notes by Tim Berners-Lee: they are not endorsed by W3C. They are aimed at the technical community, to explain reasons, provide a framework to provide consistency for for future developments, and avoid repetition of discussions once resolved.

I have found that, having started this set of notes in 1990 in the (for me) novel medium of hypertext, it has been difficult to tear free of it: my attempts to lend hierachical or serial order have been doomed to failure. Further, as ideas and these web pages have evolved, it has been important for me to be able to reorganize my thoughts, grab a new leaf, shake the tree and regard it as the root. So the reader needs to be aware of this, that each page may be an attempt to put across a given concept serially, but if you are looking for an order of concepts and subconcepts, you have as much hope as you would with words in the dictionary. I can sympathise with Ted Nelson whose Litterary Machines has "a Chapter Zero, several Chapters One, one Chapter Two, and several Chapters Three", not to mention with Ludwig Wittgenstein whose Philosophical Investigations have only paragraph numbers for structure.

The notes are in a constant state of flux, sometimes minute by minute, sometimes decade by decade. Their status varies - some have typos and spelling errors, and represent thoughts half expressed, wheras others described resolved issues which have become fundamental architectural decisions in the conceptual infrastructure of the Web. Again, something in me resists the urge to draw a line and move things from here into a "done deals" space. I try to represent accurately the status of a given page in the section above the rule at the top. Definitive documents, reviewed by W3C members and others, you will find elsewhere.

Neither have I found it easy to restrict myself to separated technical or philosophical arguments and somehow this I feel is also important, the sharpening happening, after all, where the knife meets the stone.

I did draw a line between the really old ones whose dates I couldn't necessarily even find, and which were too out of date to find themselves linked into any current discussion. Hence the brown archival section on the contents page, and the brown archived notes it points to. These are really only available for completeness of archival, and not suggested reading. The other remarks here do not apply to them.

For all its (or because of its) lax flexibility, I have personally found this space a useful one. I have used it to place opionions and explanations which I have needed to express, and have found it useful to be able to express them later to others. But also I have found it a personally useful excercie to review the state of order and disporder from time to time, part of the intuitive process of making a new step. But that is all personal use and, and for the hestiations I have just outlined, I have never felt that the whole collection has been worthy of recommeding as reading as a work in itself.

Tim BL October 1998

Read whole article...

Again and again we fall back on the folklore of the principles of good design. Sometimes I need a URI for them so this is started as collection of them. I have written about some in many places. Principles such as simplicity and modularity are the stuff of software engineering; decentralization and tolerance are the life and breath of Internet. Brian Carpenter has enumerated some principles of design of the Net [carpenter]. The third pair of ideas I have found commonly useful for the Web. I mentioned them in a keynote at WWW7 and the note on Evolvability.

This is largely "motherhood and apple pie" but it still needs a home.

Simplicity

"Keep it simple, stupid!"

Simplicity is easily to quote but often ignored in strange ways. Perhaps this is because it is the eye of the beholder.

A language which uses fewer basic elements to achieve the same power is simpler.

Sometimes simplicity is confused with 'easy to understand". For example, a two-line solution which uses recursion is a pretty simple, even though some people might find it easier to work though a 10-line solution which avoids recursion.

In XML, "Processing Instructions", those things which start with "<?" are not simple. They look simple, just an extra sort of thing in the language, but the complicate what was a very clean design of elements and attributes, and a complication in the underlying syntax is has great effect. All specifications which refer to XML processing will have to figure out what to do about processing instructions as well as elements.

Modular Design

When you design a system, or a language, then if the features can be broken into relatively loosely bound groups of relatively closely bound features, then that division is a good thing to be made a part of the design. This is just good engineering. It means that when you want to change the system, you can with luck in the future change only one part, which will only require you to understand (and test) that part. This will allow other people to independently change other parts at the same time. This is just classic good software design and books have been written about it. The corollary, the TOII is less frequently met.

Modular design hinges on the simplicity and abstract nature of the interface definition between the modules. A design in which the insides of each module need to know all about each other is not a modular design but an arbitrary partitioning of the bits. (More ...)

Being part of a Modular Design

Its is not only necessary to make sure your own system is designed to be made of modular parts. It is also necessary to realize that your own system, no matter how big and wonderful it seems now, should always be designed to be a part of another larger system.

This is often much more difficult than modularity.

Tolerance

"Be liberal in what you require but conservative in what you do"

This is the expression of a principle which applies pretty well in life, (it is a typical UU tenet), and is commonly employed in design across the Internet.

Write HTML 4.0-strict. Accept HTML-4.0-Transitional (a superset of strict).

This principle can be contentious. When browsers are lax about what they expect, the system works better but also it encourages laxness on the part of web page writers. The principle of tolerance does not blunt the need for a perfectly clear protocol specification which draws a precise distinction between a conformance and non-conformance. The principle of tolerance is no excuse for a product which contravenes a standard.

Decentralization

This is a principle of the design of distributed systems, including societies. It points out that any single common point which is involved in any operation trends to limit the way the system scales, and produce a single point of complete failure.

Centralization in social systems can apply to concepts, too. For example, if we make a knowledge representation system which requires anyone who uses the concept of "automobile" to use the term "http://www.kr.org/stds/industry/automobile" then we restrict the set of uses of the system to those for whom this particular formulation of what an automobile is works. The Semantic Web must avoid such conceptual bottlenecks just as the Internet avoids such network bottlenecks.

Test of Independent Invention

If someone else had already invented your system, would theirs work with yours?

Does this system have to be the only one of its kind? This simple thought test is described in more detail in "Evolution" in these Design Issues. It is connectted to modularity inside-out: designing a system not to be modular in itself, but to be a part of an as-yet unspecified larger system. A critical property here is that the system tries to do one thing well, and leaves other things to other modules. It also has to avoid conceptual or other centralization, as no two modules can claim the need to be the unique center of a larger system.

Principle of Least Power

In choosing computer languages, there are classes of program which range from the plainly descriptive (such as Dublin Core metadata, or the content of most databases, or HTML) though logical languages of limited power (such as access control lists, or conneg content negotiation) which include limited propositional logic, though declarative languages which verge on the Turing Complete (Postscript is, but PDF isn't, I am told) through those which are in fact Turing Complete though one is led not to use them that way (XSLT, SQL) to those which are unashamedly procedural (Java, C).

The choice of language is a common design choice. The low power end of the scale is typically simpler to design, implement and use, but the high power end of the scale has all the attraction of being an open-ended hook into which anything can be placed: a door to uses bounded only by the imagination of the programmer.

Computer Science in the 1960s to 80s spent a lot of effort making languages which were as powerful as possible. Nowadays we have to appreciate the reasons for picking not the most powerful solution but the least powerful. The reason for this is that the less powerful the language, the more you can do with the data stored in that language. If you write it in a simple declarative from, anyone can write a program to analyze it in many ways. The Semantic Web is an attempt, largely, to map large quantities of existing data onto a common language so that the data can be analyzed in ways never dreamed of by its creators. If, for example, a web page with weather data has RDF describing that data, a user can retrieve it as a table, perhaps average it, plot it, deduce things from it in combination with other information. At the other end of the scale is the weather information portrayed by the cunning Java applet. While this might allow a very cool user interface, it cannot be analyzed at all. The search engine finding the page will have no idea of what the data is or what it is about. This the only way to find out what a Java applet means is to set it running in front of a person.

I hope that is a good enough explanation of this principle. There are millions of examples of the choice. I chose HTML not to be a programming language because I wanted different programs to do different things with it: present it differently, extract tables of contents, index it, and so on.

Read whole article...

There are many other data models which RDF's Directed Labelled Graph (DLG) model compares closely with, and maps onto. See a summary in

• What the Semantic Web can represent

One is the Relational Database (RDB) model.

The Semantic Web and Entity-Relationship models

Is the RDF model an entity-relationship mode? Yes and no. It is great as a basis for ER-modelling, but because RDF is used for other things as well, RDF is more general. RDF is a model of entities (nodes) and relationships. If you are used to the "ER" modelling system for data, then the RDF model is basically an openning of the ER model to work on the Web. In typical ER model involved entity types, and for each entity type there are a set of relationships (slots in the typical ER diagram). The RDF model is the same, except that relationships are first class objects: they are identified by a URI, and so anyone can make one. Furthurmore, the set of slots of an object is not defined when the class of an object is defined. The Web works though anyone being (technically) allowed to say anything about anything. This means that a relationship between two objects may be stored apart from any other information about the two objects. This is different from object-oriented systems often used to implement ER models, which generally assume that information about an object is stored in an object: the definition of the class of an object defines the storage implied for its properties.

For example, one person may define a vehicle as having a number of wheels and a weight and a length, but not foresee a color. This will not stop another person making the assertion that a given car is red, using the color vocabular from elsewhere.

Apart from this simple but significant change, many concepts involved in the ER modelling take across directly onto the Semantic Web model.

The Semantic Web and Relational Databases

The semantic web data model is very directly connected with the model of relational databases. A relational database consists of tables, which consists of rows, or records. Each record consists of a set of fields. The record is nothing but the content of its fields, just as an RDF node is nothing but the connections: the property values. The mapping is very direct

• a record is an RDF node;
• the field (column) name is RDF propertyType; and
• the record field (table cell) is a value.

Indeed, one of the main driving forces for the Semantic web, has always been the expression, on the Web, of the vast amount of relational database information in a way that can be processsed by machines.

RDF's serialization format -- its syntax in XML -- is a very suitable format for expressing relational database information.

Special aspects of the RDB model

Relational database systems manage RDF data, but in a specialized way. In a table, there are many records with the same set of properties. An individual cell (which corresponds to an RDF property) is not often thought of on its own. SQL queries can join tables and extract data from tables, and the result is generally a table. So, the practical use for which RDB software is used typically optimized for doing operations with a small number of tables some of which may have a large number of elements.

A fundamental aspect of a database table is that often the data in a table can be definitive. Neither RDF nor RDB models have simple ways of expressing this. For example, not only does a row in a table indicate that there is a red car whose Massachusetts plate is "123XYZ", but the table may also carry the unwritten semantics that if any car has a Massachusetts plate then it must be in the table. (If any RDF node has "Massachusetts plate number" property then than node is a member of the table) The scope of the uniquenes of a value is in fact a very interest property.

The original RDB model defined by E.F. Codd included datatyping with inheritance, which he had intended would be implememnted in the RDB products to a greater extent that it has. For example, typically a person's home address house number may be typed as an an integer, and their shoe size may also be also be typed as an integer. One can as a result join to tables through those fields, or list people whose shoe size equals their house number. Practical RDB systems leave it to the application builder to only make operations which make sense. Once a database is expreted onto the Web, it becomes possible to do all kinds of strange combinations, so a stronger typing becomes very useful: it becomes a set of inference rules.

In a pure RDB model, every table has a primary key: a column whose value can be used to uniquely identify every row. Some products do not enforce this, leading to an ambiguity in the significance of duplicate rows. A curious feature is that the primary key can be changed without changing the identity of a row. (A person can change their name for example). SQL allows tables to be set up so that such changes can cascade through the local system to preseve referential integrity. This clearly won't work on the Web. One solution is to use a row ID -- which many systems do in fact use although SQL doesn't expose it in a standard way. Another is for the application to coinstrain the primary key not to change. Another is to put up with links breaking.

RDB systems have datatypes at the atomic (unstructured) level, as RDF and XML will/do. Combination rules tend in RDBs to be loosely enforced, in that a query can join tables by any columns which match by datatype -- without any check on the semantics. You could for example create a list of houses that have the same number as rooms as an employee's shoe size, for every employee, even though the sense of that would be questionable.

The new SQL99 standard is going to include new object-oriented features, such as inherited typing and structured contents of cells - arrays and structs. This RDB model with things from the OO world. I don't deal with that here in that the RDF model works as a lowest commoin denominator being able to express either and both.

Schemas and Schemas

A difference between XML/RDF schemas (and SGML) on the one hand and database schemas on the other is the expectation that there will be a relatively small number of XML/RDF schemas. Many web sites will export documents whose structure is defined by the same schema, and this is in fact what provides the interoperability.

A database schema is, as fasr as I know, created independently for each database. Even if a million companies clone the same form of employee database, there will be a million schemas, one for each database.

It may be that RDF will fill a simple role in simply expressing the equivalence of the terms in each database schema.

Exposing a database on the Web

In order to be able to access a table, and make extra statements about it which will enable its use in more and more ways, the essential objects of the table must be exported as first class objects on the Web.

When mapping any system onto the Web, the mapping into URI space is critical. Here we are doing this common operation generically for all relational databases. It is obviously usefuil for this to be done in a consistent ways between multiple vendors would be useful - an area for possible standardization.

Here is a random example I may have gotten wrong, basd on whatI understand of the naming within databases. The database itself is defined within a schema which is listed in a catalog.

2002 version, see real code implemented by Dan Connolly:

@@@ How to use typing to indicate that the URI in the table is a (relative?) URI to another object, not a string?

@@@ This works fine when implemented live on a database. However, it is a little tricky to emulate in a typical file-based web server because of the use of "personnel" in this case both as directory and as

One of the things which makes life easier is to make the mapping so that the relative URI syntax can be used to advantage. For example, here, everything within the database (the scope of an SQL statement) can be writted as a short URI.

There is a question as to how much of the SQL query syntax should be turned into identifier. For example, is a query on a primary key really an identifier? Is the extraction of a single cell really an identifier? It would be useful to be able to treat them as such. However, it would be wiser to use the "?" convention to indicate a generalized SQL idempotent query. (A URL should of course never be used to refer to the results of a table-changing operation such as UPDATE or DELETE. In this case, if HTTP were used, an SQL query should IMHO be POST ed to the database URI. Of course, you can use your favorite networked database access protocol)

In the above the column name of the table could be refered to using the table as a namespace, a row for example being

<foo
  xmlns:t="http://www.example.com/mycat/personnel/employees">
  <t:email>joe@example.com</t:email>
  <t:age>45</t:age>
</foo>

and one row of the the result of joining this table (of people) and another table (about people) by their primary keys would use namespaces from both tables:

<foo
  xmlns:t="http://www.example.com/mycat/personnel/employees"
  xmlns:u="http://www.acme.com/mycat/schema1/empdb/likes">
    <t:email>joe@example.com</t:email>
  <t:age>45</t:age>
  <u:music>blues</u:music>
</foo>

This note is an attempt to answer the question, "Why should I use RDF - why not just XML?". This has been a question which has been around ever since RDF started. At the W3C Query Language workshop, there was a clear difference of view between those who wanted to query documents and those who wanted to extract the "meaning" in some form and query that. This is typical. I wrote this note in a frustrated attempt to explain whatthe RDF model was for those who though in terms of the XML model. I later listened to those who thought in terms of the XML model, and tried to writ it the other way around in another note. This note assumes that the XML data model in all its complexity, and the RDF syntax as in RDF Model and Syntax, in all its complexity. It doesn't try to map one directly onto the other -- it expresses the RDF model using XML.

Let me take as an example a single RDF assertion. Let's try "The author of the page is Ora". This is traditional. In RDF this is a triple

triple(author, page, Ora)

which you can think of as represented by the diagram

How would this information be typically be represented in XML?

<author>
     <uri>page</uri>
     <name>Ora</name>
</author>

or maybe

<document href="page">
   <author>Ora</author>
</document>

or maybe

<document>
   <details>
    <uri>href="page"</uri>
    <author>
        <name>Ora</name>
    </author>
    </details>
</document>

or maybe

<document>
   <author>
    <uri>href="page"</uri>
    <details>
        <name>Ora</name>
    </details>
    </author>
</document>

<document href="http://www.w3.org/test/page" author="Ora" />

The XML Graph

These are all perfectly good XML documents - and to a person reading then they mean the same thing. To a machine parsing them, they produce different XML trees. Suppose you look at the XML tree

<v>
   <x>
    <y> a="ppppp"</y>
    <z>
        <w>qqqqq</w>
    </z>
   </x>
</v>

It's not so obvious what to make of it. The element names were a big hint for a human reader.

Without looking at the schema, you know things about the document structure, but nothing else. You can't tell what to deduce. You don't know whether ppppp is a y of qqqqq, or qqqqq is a z of ppppp or what. You can't even really tell what real questions can be asked. A source of some confusion is that in the xyz example above, there are lots of questions you can ask. They are questions like,

• Is there a w element within a details element?
• What is the content of the w element within the first x element?
• What is the content of the w element following the first y element which contains an x element whose a attribute is "pppp"?
• and so on.

These are all questions about the document. If you know the document schema (a big if) , and if that schema it only gives you a limited number of ways of expressing the same thing (another big if) , then asking these questions can be in fact equivalent to asking questions like

• What is the author of page?

This is hairy. It is possible because there is a mapping from XML documents to semantic graphs. In brief, it is hairy because

• The mapping is many to one
• You need a schema to know what the mapping is
• (The schemas we are talking about for XML at the moment do not include that anyway and would have to have a whole inference language added)
• The expression you need for querying something in terms of the XML tree is necessarily more complicated than the expression you need for querying something in terms of the RDF tree.

This last is a big one. If you try to write down the expression for the author of a document where the information is in some arbitrary XML schema, you can probably do it though it may or may not be very pretty. If you try to combine more than one property into a combined expression, (give me a list of books by the same author as this one), saying it in XML gets too clumsy to consider.

(Think of trying to define the addition of numbers by regular expression operations on the strings. Its possible for addition. When you get to multiplication it gets ridiculous - to solve the problem you would end up reinventing numbers as a separate type.)

Looking at the simple XML encoding above,

<author>
     <uri>page</uri>
     <name>Ora</name>
</author>

it could be represented as a graph

We can represent the tree more concisely if we make a shorthand by writing the name of each element inside its circle:

Of course the RDF tree which this represents (although it isn't obvious from the XML tree except to those who know) is

Here we have made a shorthand again by putting making the label for each part its URI.

The complexity of querying the XML tree is because there are in general a large number of ways in which the XML maps onto the logical tree, and the query you write has to be independent of the choice of them. So much of the query is an attempt to basically convert the set of all possible representations of a fact into one statement. This is just what RDF does. It gives you some standard ways of writing statements so that however it occurs in a document, they produce the same effect in RDF terms. The same RDF tree results from many XML trees.

Wouldn't it be nice if we could label our XML so that when the parser read it, it could find the assertions (triples) and distinguish their subjects and objects, so as to just deduce the logical assertions without needing RDF? This is just what RDF does, though.

The RDF Graph

In fact RDF is very flexible - it can represent this triple in many ways in XML so as to be able to fit in with particular applications, but just to pick one way, you could write the above as

<Description about="http://www.w3.org/test/page" Author ="Ora" />

I have missed out the stuff about namespaces. In fact as anyone can create or own the verbs, subjects and objects in a distributed Web, any term has to be identified by a URI somehow. This actual real example works out to in real life more like

<?xml version="1.0"?>
  <Description

          xmlns="http://www.w3.org/TR/WD-rdf-syntax#"
          xmlns:s="http://docs.r.us.com/bibliography-info/"
 
                  about="http://www.w3.org/test/page" 
                  s:Author ="http://www.w3.org/staff/Ora" />

You can think that the "description" RDF element gives the clue to the parser as to how to find the subjects, objects and verbs in what follows.

This is pretty much the most shorthand way of using the base RDF in XML. There are others which are longer, but more efficient when you have, for instance, sets of many properties of the same object. The useful thing is that of course they all convey the same triple

It is a mess when you use questions about a document to try to ask questions about what the document is trying to convey. It will work. In a way. But flagging the grammar explicitly (RDF syntax is a way of doing this) is a whole lot better.

Things you can do with RDF which you can't do with XML include

• You can parse the semantic tree, which end up giving you a set of (possibly mutually referential) triples and then you can use the ones you want ignoring the ones you don't understand.

Problems with basing you understanding on the structure include

• Without having gone to the trouble of getting the schema, or having an application hand-programmed to recognise a particular document type, you can't pick up any semantic information from a document;
• When an XML schema changes, it could typically introduce new intermediate elements (like "details" in the tree above or "div" is HTML). These may or may or may not invalidate any query which has been based on the structure of the document.
• If you haven't gone to the trouble of making a semantic model, then you may not have a well defined one.

I'll end this with some examples of the last problem. Clearly they can be avoided by good design even in an XML system which does not use RDF. Using RDF makes things easier.

Get it right

If you haven't gone to the trouble of making a semantic model, then you may not have a well defined one. What does that mean? I can give some general examples of ambiguities which crop up in practice. In RDF, you need a good idea about what is being said about what, and they would tend not to arise.

Look at a label on the jam jar which says: "Expires 1999". What expires: the label, or the jam? Here the ambiguity is between a statement about a statement about a document, and a statement about a document.

Another example is an element which qualifies another apparently element. When information is assembled in a set of independently thrown in records often ambiguities can arise because of the lack of logic. HTTP headers (or email headers) are a good example. These things can work when one program handles all the records, but when you start mixing records you get trouble. In XML it is all too easy to fall into the trap of having two elements, one describing the author, and a separate one as a flag that the "author" element in fact means not the direct author but that of a work translated to make the book in question. Suddenly, the "author" tag, which used to allow you to conclude that the author of a finnish document must speak finnish, now can be invalidated by an element somewhere else on the record.

Another symptom of a specification where the actual semantics may not be as obvious as as first sight is ordering. When we hear that the order of a set of records is important, but the records seem to be defined independently, how can that be? Independent assertions are always valid taken individually or in any order. In a server configuration file, for example, he statement which looks like "any member has access to the page" might really mean "any member has access to the page unless there is no other rule in this file which has matched the page". That isn't what the spec said, but it did mention that the rules were processed in order until one applied. Represented logically, in fact there is a large nested conditional. There is implicit ordering when mail headers say, "this message is encrypted", "this message is compressed", "this message is ASCII encoded", "this message is in HTML". In fact the message is an ASCII encoded version of an encrypted version of a compressed version of a message in HTML. In email headers the logic of this has to be written into the fine print of the specification.

Order in documents

There is something fundamentally different between giving a machine a knowledge tree, and giving a person a document. A document for a person is generally serialized so that, when read serially by a human being, the result will be to build up a graph of associations in that person's head. The order is important.

For a graph of knowledge, order is not important, so long as the nodes in common between different statements are identified consistently. (There are concepts of ordered lists which are important although in RDF they break down at the fine level of detail to an unordered set of statements like "The first element of L is x", the "third element of L is z", etc so order disappears at the lowest level.). In machine-readable documents a list of ostensibly independent statements where order is important often turn out to be statements which are by no means independent.

Some people have been reluctant to consider using an RDF tree because they do not wish to give up the order, but my assumption is that this is from constraints on processing human readable documents. These documents are typically not ripe for RDF conversion anyway.

Conclusion:

Sometimes it seems there is a set of people for whom the semantic web is the only graph which they would consider, and another for whom the document tree (or graph if you include links) is all they would consider. But it is important to recognise the difference.

What the Semantic Web can represent

There are many other data models which RDF's Directed Labelled Graph (DLG) model compares closely with, and maps onto. This page is written with the intention of enumerating the similarity and diferences between the models, to indicate how the mapping might be done and what extra information muast be added in the process. Where the other models are related to previous unmet promises of computer science, now passed into folk law as unsolvable problems, they suggest a fear that the goal of a Semantic Web is inappropriate.

One consistent difference between the Semantic Web and many data models for programming langauges is the "closed world assumption".

A Semantic Web is not Artificial Intelligence

The concept of machine-understandable documents does not imply some magical artificial intelligence which allows machines to comprehend human mumblings. It only indicates a machine's ability to solve a well-defined problem by performing well-defined operations on existing well-defined data. Instead of asking machines to understand people's language, it involves asking people to make the extra effort

Even though it simple to define, RDF at the level with the power of a semantic web will be complete language, capable of expressing paradox and tautology, and in which it will be possible to phrase questions whose answers would to a machine require a search of the entire web and an unimaginable amount of time to resolve. This should not deter us from making the language complete. Each mechanical RDF application will use a schema to restrict its use of RDF to a deliberately limited language. However, when links are made between the RDF webs, the result will be an expression of a huge amount of information. It is clear that because the Semantic Web must be able to include all kinds of data to represent the world, tha the language itself must be compeletely expressive

A semantic Web will not require every application to use expressions of arbitrary complexity

Even though the language itself allows expressions of arbitrary complexity and computability, applications which generate RDF will in practice be limited to generating simple expressions such as access control lists, privacy preferences, and search criteria. This does not mean that where a "not" is needed, it should not be drawn from a standard vocabulary so than any RDF engine will be able to recognise it as a "not".

(more)

A semantic Web will not require proof generation to be useful: proof validation will be enough.

The first uses, such as access control on web sites, involve validation of a previously prepared proof, not a requirement to answer an arbitrary question, find the path the construct a valid proof. It is well known that to search for and generate a proof for an arbitrary question is typically an intractable process for many real world problems, and RDF does not require this (unsolvable) problem to be solved to be useful.

A semantic web is not an exact rerun of a previous failed experiment

Other concerns at this point are raised about the relationship to Knowledge representation systems: has this not been tried before with projects such as KIFand cyc? The answer is yes, it has, more or less, and such systems have been developed a long way. They should feed the semantic Web with design experience and the Semantic Web may provide a source of data for reasoning engines developed in similar projects.

Many KR systems had a problem merging or interrelating two separate knowledge bases, as the model was that any concept had one and only one place in a tree of knowledge. They therefore did not scale, or pass the test of independent invention. [see evolvability]. The RDF world, by contrast is designed for this in mind, and the retrospective documentation of relationships between originally independent concepts.

Knowledge Representation goes Global

Knowledge representation is a field which is currently seems to have the reputation of being initially interesting, but which did not seem to shake the world to the extent that some of its proponents hoped. It made sense but was of limited use on a small scale, but never made it to the large scale. This is exactly the state which the hypertext field was in before the Web. Each field had made certain centralist assumptions -- if not in the philosophy, then in the implementations, which prevented them from spreading globally. But each field was based on fundamentally sound ideas about the representation of knowledge. The Semantic Web is what we will get if we perform the same globalization process to Knowledge Representation that the Web initially did to Hypertext. We remove the centralized concepts of absolute truth, total knowledge, and total provability, and see what we can do with limited knowledge.

The Semantic Web and Entity-Relationship models

Is the RDF model an entity-relationship mode? Yes and no. It is great as a basis for ER-modelling, but because RDF is used for other things as well, RDF is more general. RDF is a model of entities (nodes) and relationships. If you are used to the "ER" modelling system for data, then the RDF model is basically an opening of the ER model to work on the Web. In typical ER model involved entity types, and for each entity type there are a set of relationships (slots in the typical ER diagram). The RDF model is the same, except that relationships are first class objects: they are identified by a URI, and so anyone can make one. Furthurmore, the set of slots of an object is not defined when the class of an object is defined. The Web works though anyone being (technically) allowed to say anything about anything. This means that a relationship between two objects may be stored apart from any other information about the two objects. This is different from object-oriented systems often used to implement ER models, which generally assume that information about an object is stored in an object: the definition of the class of an object defines the storage implied for its properties.

For example, one person may define a vehicle as having a number of wheels and a weight and a length, but not foresee a color. This will not stop another person making the assertion that a given car is red, using the color vocabulary from elsewhere.

Apart from this simple but significant change, many concepts involved in the ER modelling take across directly onto the Semantic Web model.

The Semantic Web and Relational Databases

The semantic web data model is very directly connected with the model of relational databases. A relational database consists of tables, which consists of rows, or records. Each record consists of a set of fields. The record is nothing but the content of its fields, just as an RDF node is nothing but the connections: the property values. The mapping is very direct

• a record is an RDF node;
• the field (column) name is RDF propertyType; and
• the record field (table cell) is a value.

Indeed, one of the main driving forces for the Semantic web, has always been the expression, on the Web, of the vast amount of relational database information in a way that can be processsed by machines.

RDF's serialization format -- its syntax in XML -- is a very suitable format for expressing relational database information.

Relational database systems, manage RDF data, but in a specialized way. In a table, there are many records with the same set of properties. An individual cell (which corresponds to an RDF property) is not often thought of on its own. SQL queries can join tables and extract data from tables, and the result is generally a table. So, the practical use for which RDB software is used typically optimized for soing operations with a small number of tables some of which may have a large number of elements.

RDB systems have datatypes at the atomic (unstructured) level, as RDF and XML will/do. Combination rules tend in RDBs to be loosely enforced, in that a query can join tables by any comlumns which match by datatype -- without any check on the semantics. You could for example create a list of houses that have the same number as rooms as an employee's shoe size, for every employee, even though the sense of that would be questionable.

The Semantic Web is not designed just as a new data model - it is specifically appropriate to the linking of data of many different models. One of the great things it will allow is to add information relating different databases on the Web, to allow sophisticated operations to be performed across them.

RDF is not an Inference system

I am not proposing any FPOC or HOL inference engine. I just note that HOL allows integration of multiple systems which use different inference engines spanning the range from from SQL to AI. For example, a simple HOL would allow any SHOE rules, data and results expressed, and a proof found by a SHOE engine to be verified by anyone.

Surely all first-order or higher-order predicate caluculus based systems (such as KIF) have failed historically to have wide impact?

The same was true of hypertext systems between 1970 and 1990, ie before the Web. Indeed, the same objection was raised to the Web, and the same reasons apply for pressing on with the dream.

The problem with all such systems was that they were conceptually or physically centralized. They required link global consistency.

Guess what? KIF is very centralized in its approach to organizing knowledge (the cyc ontology for example suggests that everyone agree on the same terms for common english words, which RDF does not) and it does not promote its concepts to being first class web objects, ie it doesn't use URIs to identify them. To webize KIF or KR in general is, in many ways, the same as to webize hypertext in many ways. Replace identifiers with URIs. Remove any requirement for global consistency. Put in a significant effort into getting critical mass. Sit back.

Surely, many things expressible in FOPC are not efficiently computable?

Dead right. The goal of the semantic web is to express real life. Many things in real life, real questions which we will face are not efficiently computable. There are two solutions to this: The classical (pre-web) solution is to constrain the langage of expression so that all queries terminate in finite time. The weblike solution is to allow the expression of facts and rules in an overall language which is sufficiently flexible and powerful to express real life. Create subsets fo the web in which specific constraints give you specific computational properties. An anlogy is with the human-information systems which existed before the web. Most forced one to keep ones data in a hierarchy (sometimes of fixed depth or a matrix (often with a specific number of dimensions). This gave consistency properties within the information system. I bet DARPA has many of these systems and still does. They only way they could be integrated was to express them in terms of a much more powerful language - global hypertext. Hypertext did not have any of these reassuring properties. People were frightened about getting lost in it. You could follow links forever. As it turns out, it is true of course that there is a problem that you can follow links forever in the Web. And on the Semantic Web an inference engine will not necessarily terminate. However, on eth Web there are many subsystems such as many websites where life is very ordered and predictable, and searches give definitive results and there are no dangling links. But there is a HUGE advantage from exposing all this information in a way that allows it to be unified with all the other systems, ordered and unordered.

We should not expect a base inference level to include non-decidable computations

I have no expecatation of any inference capability in the SW core design. The semantic web does not have HOL inference as a standard. I would expect any SW compliant device to be able to validate a HOL proof, but not generate one.

If you take a non-HOL-complete langauge and extend it to HOL, unless you have first defined where you are going (by defininbg the HOL langauge and expressing SHOE in it first) you will very likely end up with a rather baroque HOL langauge.

The FOPC inference model is extremely intolerant of inconsistency [i.e. P(x) & NOT (P(X)) -> Q], the semantic web has to tolerate many kinds of inconsistency.

Toleration of inconsistecy can only be done by fuzzy systems. We need a semantic web which will provide guarantees, and about which one can reson with logic. (A fuzzy system might be good for finding a proof -- but then it should be able to go back and justify each deduction logically to produce a proof in the unifying HOL language which anyone can check) Any real SW system will work not by believing anything it reads on the web but by checking the source of any information. (I wish people would learn to do this on the Web as it is!). So in fact, a rule will allow a system to infer things only from statements of a particular form signed by particular keys. Within such a system, an inconsistency is a serious problem, not something to worked around. If my bank says my bank balance is $100 and my computer says it is $200, then we need to figure out the problem. Same with launching missiles, IMHO. The semantic web model is that a URI dereferences to a document which parses to a directed labeled graph of statements. The statements can have URIs as prameters, so they can may statements about documents and about other statements. So you can express trust and reason about it, and limit your information to trusted consistent data.

Again, extension to higher order logic makes sense to me, requirement of FOPC inference model seems dangerous.

Most KR systems confuse information with inference tips. When a system stores a rule a daughter of one's daughter is one's grandaughter it is typically not just tored as that statement, but in a table of rules to be used by the algorithm at a particular time (for example whenever a parent of a daughter is found). The classicfication between data and various type of rule is a sort of meta level information which is general not itself expressed in the language. Two systems must be able to interchange the logical meaning of the rule, even when the type of rule may be unknown to each others inference engines. (Of couse, the rule expressed in general logic may be recongizable as a rule by another system and absorbed as such.) The example above is logically

∀α,β,χ (d(a,b) & d(b,c) => gd(a,c))

while for example a SHOE-based system and an Algernon-based system may have quite different systems for applying rules at different times.

Conceptual Graphs and the Semantic Web

Reification in this context means the expression of something in a language using the language, so that it becomes treatable by the language. RDF graphs consist of RDF statements. If one wants to look objectively at an RDF graph and reason about it is using RDF tools, then it is useful, at least in theory, to have an ontology for describe RDF statements. This note described one suitable ontology.

When RDF extended to N3, then one way of discussing the semantics is to describe N3 documents in RDF. This document does both.

The namespace used is <http://www.w3.org/2004/06/rei#> , for which here we use the rei: prefix. Also, we use the ex: prefix for the namespace <http://example.com/ex#>.

RDF Terms

RDF terms are nodes in the RDF Graph. In RDF, these can be of three types: named nodes, blank nodes, and literals. We will also call named nodes symbols.

Symbols

Named nodes are named by URI strings, so a named node can be defined simply by its URI string. The symbol which in N3 is written as <http://example.com/> would be described as the RDF node:

[ a rei:Symbol;  rei:uri "http://example.com/ex#joe" ]

Blank nodes

Blank nodes (or Bnodes for short) are nodes do not have URIs. When describing a graph, we can say that a node is blank by saying that it is in the class rei:BNode.

[ a rei:Bnode ]

This blank node in the description is a description of a blank node in the original graph. They are node the same blank node. We could in fact name the blank node for the purposes of description:

ex:bnode1 a rei:BNode.

Literals

Literals in an RDF graph are defined only by their value, just as symbols are defined by their URIs. When using RDF to describe RDF, RDF literals can clearly be used to give the value:

[ a rei:Literal, rei:value "The quick brown fox"]

In fact, the domain of rei:value is rei:Literal, so it is not necessary to explicitly state that something is a literal, one can just write:

[rei:value "The quick brown fox"]

RDF Statements

A RDF statement is defined by its three parts, known as subject, predicate and object, each of which is a term. In RDF, neither the subject nor the predicate may be a Literal. The statement which in N3 is ex:joe ex:name "James Doe". would be described as

[ a rei:Statement;
  rei:subject [rei:uri "http://example.com/ex#joe"];
  rei:predicate [rei:uri "http://example.com/ex#name"];
  rei:object [rei:value "James Doe"]
] 

In fact, the fact that it is a rei:Statement would have been clear as the domains of rei:subject, rei:predicate and rei:object are all rei:Statement.

RDF Graphs

An RDF graph is a set of statements. RDF itself doesn't have the concept of a set, it only has the concept of an ordered list (RDF collection). However, the OWL relation owl:oneOf related a class to a list of its members, and so we can form a set the set containing 3 4 and 5 as [ owl:oneOf (3 4 5)] . using this convention, we can describe an RDF Graph as the set of statements. For example, the graph whose contents which would be written, in N3 as

ex:joe ex:name "James Doe".
ex:jane ex:name "Jane Doe".

would be described in this ontology as:

{  a rei:RDFGraph;
   statements [ owl:oneof (
      [ a rei:Statement;
  rei:subject [rei:uri "http://example.com/ex#joe"];
  rei:predicate [rei:uri "http://example.com/ex#name"];
  rei:object [rei:value "James Doe"]
      ]
      [ a rei:Statement;
  rei:subject [rei:uri "http://example.com/ex#jane"];
  rei:predicate [rei:uri "http://example.com/ex#name"];
  rei:object [rei:value "Jane Doe"]
      ] ) 

Using the set may be ungainly, but it ensures that two RDFGraphs which contain the same statements are demonstrably the same in their reified form. (We envisage that further developments systems may have explicit processing for sets, and N3 syntax could even be extended to include set literal syntax, which would of course make this easier.)

The quoting of URIs

The use of an explicit string as the URI for the subject above is also ungainly, compared with the use in the original N3 where a prefixed symbol can be used. Why is the string given explicitly, instead of writing it as symbol?

Let's suppose for a moment that we just use the symbol, not the string for the URI:

#Wrong:
[ a rei:Statement;
  rei:subject ex:joe;
  rei:predicate ex:name;
  rei:object [rei:value "James Doe"]
] 

This should be a description of an RDF statement. It must preserve the original graph, including the URIs it used. The statements which would be described as

[ rei:subject ex:joe;                        # Wrong
  rei:predicate ex:name;
  rei:object [rei:value "James Doe"]] 

and

[ rei:subject ex:jd1;                         # Wrong
  rei:predicate ex:name;
  rei:object [rei:value "James Doe"]] 

are different graphs, even if "http://example.com/ex#joe" and "http://example.com/ex#jd1" are two URIs for the same person. However, if the system knows that <ex:jd1> and <ex:joe> are in fact thhe same person, then the second statement can be derived from the first. It is important (in our application) to be able to know which name a graph used for something. The form of reification which is provided by the original RDF specification is not suitable, because it loses that information.

N3 Formulae

N3 extends RDF to allow graphs themselves to be another form of literal node. A graph can be quoted inside another graph, as one of the terms of a statement:

ex:jane  ex:knows   { ex:joe ex:name  "James Doe" }.

Jane knows "joe's name is 'James Doe'". As above, the quotation effect is important. Jane's knowledge is in these terms. Even though ex:jd1 and ex:joe may be the same person, Jane might not know that, and so may not know that ex:jd1's name is James Doe.

An N3 formula also introduces quantification. Variables are introduced by allowing a given set of symbols to be universally quantified over the formula, and another set to be universally quantified.

A formula is described by three sets: the set of statements (the graph), the set of universals and the set of existentials. The semantics of an N3 formula are that the universal quantification is applied to the result of applying the existential quantification to the conjunction of the statements. (a la forall x: exists c such that ...). The N3 formula

  @keywords a.
  [] a car. { ?x a car } => { ?x a vehicle }.

(roughly, There is a car. Anything which is a car is a vehicle) is shorthand for

@keywords a.
@forAll :x.
  @forSome :c.
    :c a car.
    {x a car } => {x a vehicle}.

@prefix : <http://www.w3.org/2004/06/rei#> .
@prefix owl: <http://www.w3.org/2002/07/owl#> .
@keywords a.
    
[ a <http://www.w3.org/2000/10/swap/log#Truth>;
  universals[owl:oneOf(
                "http://www.w3.org/2000/10/swap/test/reify/ex1.n3#x"  ) ];
  existentials [owl:oneOf(
                "http://www.w3.org/2000/10/swap/test/reify/ex1.n3#c"  ) ];
  statement [ owl:oneOf([
     object [uri "http://www.w3.org/2000/10/swap/test/reify/ex1.n3#car" ];
     predicate [uri "http://www.w3.org/1999/02/22-rdf-syntax-ns#type" ];
     subject [uri "http://www.w3.org/2000/10/swap/test/reify/ex1.n3#c" ]
               ]  [
     object [
         universals [ owl:oneOf () ] ];
         existentials [ owl:oneOf () ];
         statements [ owl:oneOf (
           [ object  [uri "http://www.w3.org/2000/10/swap/test/reify/ex1.n3#vehicle" ];
             predicate [uri "http://www.w3.org/1999/02/22-rdf-syntax-ns#type" ];
             subject[ uri "http://www.w3.org/2000/10/swap/test/reify/ex1.n3#x" ] ] ) ];
     predicate [uri "http://www.w3.org/2000/10/swap/log#implies" ];
     subject  [
         universals  [owl:oneOf ()];
         existentials [owl:oneOf () ];
         statements  [owl:oneOf  (
           [ object [uri "http://www.w3.org/2000/10/swap/test/reify/ex1.n3#car" ];
             predicate [uri "http://www.w3.org/1999/02/22-rdf-syntax-ns#type" ];
             subject [uri "http://www.w3.org/2000/10/swap/test/reify/ex1.n3#x" ] ] ) ]] ] ) ] ].
    

Asserting truth

Note that in this mode, the formula is not only described, but it is also stated to be a Truth. To simply describe a formula as existing doesn't say anything. Formulae are abstract things, to say one exists doesn't add anything. Some would say, all formulae exist, just as all lists exist. However, to assert that one is true asserts its contents. The RDF file output above has, by definition of the terms in the reification namespace, the same meaning as the full N3 formula from which it is produced. It does to any agent which understands the meaning of the reification namespace.

URIs are so ubiquitous that they are now used in many different sorts of system. There is a whole class of issues which arise when bits of the web architecture are designed by those who are not aware of the ways in which others use them, an end up coding or speccing so as to break others' preferred design patterns. One of these patterns is the use of relative URIs.

Recap: Local IDs

Localidentifiers as an important case of URIs. Elsewhere in the these notes there are more complete descriptions of the way the URI system works, and specifically as to how the # character is used to take what was locally only a local identifier and by prepending the URI of the document in which that local identifier is used, turning it into a global identifier. Hence, hypertext turns into the WWW, and data files turn into the Web of Data, and so on. We call it webizing a system. Here is web architecture in one line:

 
            <global id of document>  #  <local id of whatever>
 

So when a file on the web refers to something internal it can just use the local ID. It would be very cumbersome to include the whole URI of the document itself in each of these references.

 
            #  <local id of whatever>
 

• This file can be copied and put up on any web server
• The file can therefore be used as a template for other new systems
• The fie is shorter and much easier to read than one in which all the URIS are spelled out in full.

     pi = 3.14159265359;
     print (2 * pi);

    <file:///users/foo/programs/play/circles.py#pi> = 3.14159265359;
    print (2 * <file:///users/andys/programs/play/circles.py#pi>);

Relative URIs

In fact, when a number of files all refer to each other such as a bunch of HTML, CSS, SVG and JS files as part of w web application, or a set of Turtle files as part of a Linked Data dataset, then also save a lot of space and fragility by using relative URIs, using the unix-like conventions

There are, though, relatively few systems which in fact can be implemented by a single file. There are a huge number in which the system is implemented in one directory of a file system, or in that and its descendants in the tree. For example, an HTML file may have local CSS files and images. The system may be edited in unix file space, where any URLs will be file:/// URLs, and it may at the same time by seen through a web server, where the various resources are referred to by relative URLs:

    <a href="intro.html">
     <img src="../images/foo.png"/>
    </a>
 

It isn't just coincidence that the syntax of URLs matches that of unix-like file systems. It is designed to let systems be looked at either with file:// glasses or with http:// glasses and still work. This is very useful. This means that you can directly export onto the web a system which has been built up as a local filesystem, and it means that when you have a file-mapped web server, like Apache, you can use all kinds of unix tools to build or analyze your stuff. For anyone who has been working this way for more a decade to two this may be blindingly obvious, but if you work in a content-management system which is not a file-system-mapped one, or you work with a file system which does not use the *ix conventions, then this may not be second nature.

What the Relative URI Pattern gives you then, is the ability for you system to work with different base addresses without being modified.

• You might want to view a directory both locally as a file and also remotely through a web server.
• You might want to develop the system on a test system, with testing of the internal relationships, then move it to a production system.
• You might want a system to be visible as part of many web sites, for example a common login system or help system shared my many related websites
• You might provide parts of a web site by proxying another. Maybe when it comes to editing files your main web server proxies though to a specialist server.
• You may want a build a system which you can hand to others to copy elsewhere.

Many systems, including most of the ones I have built and use day to day, have many internal relative links but really are better built without a knowledge of their own base URI, in that stored URIs are always relative. Even if the software absolutizes them in processing them, as there are no absolute URIs for the local identifiers. This is not to say all systems are like this, but some are and they are an important class.

Note other reasons for relative URIs include readability, and storage space and transmission length.

Best Practice

Because an application developer is very likely to find it valuable to use relative URIs, any software libraries which serialize web file formats such as HTML and Turtle must provide the option (or the default) to serialize using relative URIs.

When data is stored in files, for example on a web server, it is good practice to store it using relative URIs.

When data is sent across the net, also, such as in HTTP, relative URIs should be used.

Yes, there are cases when people want to design systems without this properties, so absolute URIs should be an option.

Example

All the same-domain URIs within the system need to be relative.

Attempting to introduce Jena-based code to this system is currently blocked on the need for this bug to be resolved in Jena.

For this reason, for example cwm's serializers use relative URIs, and even the RDF/XML one has an option to put relative URIs in namespaces. (The spec officially forbids relative URIs in the namespace part of XML. This is a bug with XML. The rdflib.js library uses relative URIs by default.

Failure example

The XML space when it defines how namespaces are declared in the top of the document, officially forbids relative URLs. The authors envisages a narrow set of use cases in which the terms defined in the namespaces were all absolutely references and stable and global in nature, and the data in the document could be local and unstable. Unfortunately they did not understand the general need to generate relative URIs in arbitrary situations. The cwm command line and the cwm serializer have an option to override this constraint and generate relative URIs. A solution of course for RDF users is instead of RDF/XML to use the Turtle (nor N3) serialization, which is normally preferable anyway for many reasons.

The Jena RDF system does not (currently 2015) include code to generate relative URIs, and when it serializes data into RDF/XML or turtle, it only generates absolute URIs. This prevents it being used for a wide range of systems which require the use of relative URIs.

There is a classic failure mode for RDF systems in which developers bring up a system on test.acme.com and then move it to production.acme.com and all the links break. I know there are cases for absolute URIs but the relative URIs are very important best practices.

At at attempt to explain explain part of the relationship between the Semantic Web and inference engines, either existing or legacy, and discuss the relationship between inference rules and logical facts.

The Semantic Web is a universal space for anything which can be expressed in classical logic. In the world of knowledge Representation (KR) there are many different systems, and the following is an attempt to generalize.

Each system typically has a distinction between data and rules. The data is a pool of information in one language (sometimes very simple without negation like basic RDF) . The rules control the inference steps which the inference engine makes. The rules are written in a restricted language so as to preserve some property computability property. Algernon restricts its rules to forward chaining but assures Socratic completeness.

When integrating rules with the semantic web, one must realize that a rule contains two separate pieces of information. Take a rule in a certain inference system

g(a,c) |= d(a,b) & d(b,c)

which is defined to mean "whenever you find a new relationship where any a is the daughter of some b, then if for that b there is any c for which b is the daughter of c, then conclude that a is the granddaughter of c". Here, "conclude" means add to the database. This is a procedural instruction.

It involves an out-of band decision (may by a person) as to whether all granddaughter relationships should be added to the database the moment they can be, or whether the relationship would only be used at a time when a query is made. This rule can be exchanged between two inference engines of the same type, but it does not as a rule make sense to anyone else.

In fact, of course, this rule would be nonsense if it were not for the fact in classical logic that

Va,b,c g(a,c) <= d(a,b) & d(b,c)

This fact, unlike the rule, can be directly expressed in the semantic web language. When the rule is used in deducing something, it is this fact which is a step input to the proof. Every semantic web proof validator will be able to handle it.

Exposing rules as classic logic facts strips the (pragmatically useful) hint information which controls the actual sequence of operation of a local inference engine. When the facts corresponding to all the rules of all the inference engines are put onto the web, then the great thing is that all the knowledge is represented in the same space. The drawback is that there is no one inference engine which can answer arbitrary queries. But that is not a design goal of the semantic web. The goal is to unify everything which can be expressed in classical logic (including more mathematics when we get to it) without futher constraint. We must be able to describe hte world, and our hopes and needs and terms and conditions. A system which tries to constrain the expressive power cannot be universal.

Non-monotonic "logics"

Now there are some systems which in fact use classical logic directly, and others, "non-monotonic logics" in which adding a fact can change something which was previously "believed true" to being "believed false". (Describing them as logics may be regarded by some as questionable). For example, given that "birds can fly", the system will believe that Pingu can fly because Pingu is a penguin and a penguin is a bird, unltill it is told that penguins can't fly. Then it will assume that all birds can fly excpt for penguins. Such systems use concepts of "defaults" -- things to be assumed unless one is told otherwise. They are fundamentally closed-world systems, in that the concept of "belief" is alway implicitly make with respect to a given closed set of facts.

One can export such information into the semantic web in two ways. One can export the rule system specifically, ending up with a statement of the form "there is as assertion of birds being able to fly which is is unchallenged in the xxxx corpus by any assertion contradicting that which applied to birds or any otehr superclass of penguins". This effectivly is a reification of the non-monotonic system, an analysis not of penguins but of the inferenc system and what its state is. This may be so unweildly that it is only useful by systems which use th same inference system. The second way to export the data is to just record the classical logic statement as the output of the inference engine. "The xxxx system has output that Pingu can fly.". In certian cases, a system might risk incorporating such statements into a classic inference system. This is the logical equivalent of declaring, "Well, I don't think such a book exists becase it wasn't in Blackwell's catalog". We do things all the time, but a secure system is unlikely to be set up to incorporate such information. (A more secure system would for example, given the publisher and year, find a definitive list from the publisher of books published in that year, which would allow it to proove that such a book did not exist.)

The choice of classical logic for hte Semantic web is not an arbitrary choice among equals. Classical logic is the only way that inference can scale across the web. There are some logics which simply do not have a consistent set of axioms - fuzzy logic, for example, tends to believe something to a greater extent as a funcion of how often evidence for it has been presented. Closed world systems don't scale because the refernce to the scope of a defualt is typically implicit, and different from one fact to another. When a fact is presented as a fact, the "Oh yeah?" function of demanding justification can be satsfifed by a roof in a universal language of proof. non-classical heuristic systems may have been used to discover the proof, but onec the proof has been found it can by checked as valid by any semantic web system.

In the diagram, I have put heuristic systems above the semnatic web bus, and classical systems below. In Weaving the Web later chapters I try to describe the importanc of the web in supporting both types of system.

Up to Design Issues

Tim BL

Up to Design Issues

Tim BL

https://camps.org/campsunshine/families/smith/jane/stuff

https://jane.smith.families.campsunshine.camps.org/stuff

https://camps.org/board/AcmeDiscss/stuff

https://camps.org/board/papers

https://AcmeDiscss.camps.org/

https://AcmeDiscss.board.camps.org/

https://campsboard.AcmeDiscss.com/

This is the 2nd of 4 related notes:

1. "HTTPS Everywhere" considered harmful
2. The Same Origin Policy - Origin Granularity
3. Model Real Trust
4. Client-Side Certificates

Up to Design Issues

Tim BL

A road map for the future, an architectural plan untested by anything except thought experiments.

This was written as part of a requested road map for future Web design, from a level of 20,000ft. It was spun off from an Architectural overview for an area which required more elaboration than that overview could afford.

Necessarily, from 20,000 feet, large things seem to get a small mention. It is architecture, then, in the sense of how things hopefully will fit together. So we should recognize that while it might be slowly changing, this is also a living document.

This document is a plan for achieving a set of connected applications for data on the Web in such a way as to form a consistent logical web of data (semantic web).

Introduction

The Web was designed as an information space, with the goal that it should be useful not only for human-human communication, but also that machines would be able to participate and help. One of the major obstacles to this has been the fact that most information on the Web is designed for human consumption, and even if it was derived from a database with well defined meanings (in at least some terms) for its columns, that the structure of the data is not evident to a robot browsing the web. Leaving aside the artificial intelligence problem of training machines to behave like people, the Semantic Web approach instead develops languages for expressing information in a machine processable form.

This document gives a road map - a sequence for the incremental introduction of technology to take us, step by step, from the Web of today to a Web in which machine reasoning will be ubiquitous and devastatingly powerful.

It follows the note on the architecture of the Web, which defines existing design decisions and principles for what has been accomplished to date.

Machine-Understandable information: Semantic Web

The Semantic Web is a web of data, in some ways like a global database. The rationale for creating such an infrastructure is given elsewhere [Web future talks &c] here I only outline the architecture as I see it.

The basic assertion model

When looking at a possible formulation of a universal Web of semantic assertions, the principle of minimalist design requires that it be based on a common model of great generality. Only when the common model is general can any prospective application be mapped onto the model. The general model is the Resource Description Framework.

See the RDF Model and Syntax Specification

Being general, this is very simple. Being simple there is nothing much you can do with the model itself without layering many things on top. The basic model contains just the concept of an assertion, and the concept of quotation - making assertions about assertions. This is introduced because (a) it will be needed later anyway and (b) most of the initial RDF applications are for data about data ("metadata") in which assertions about assertions are basic, even before logic. (Because for the target applications of RDF, assertions are part of a description of some resource, that resource is often an implicit parameter and the assertion is known as a property of a resource).

As far as mathematics goes, the language at this point has no negation or implication, and is therefore very limited. Given a set of facts, it is easy to say whether a proof exists or not for any given question, because neither the facts nor the questions can have enough power to make the problem intractable.

Applications at this level are very numerous. Most of the applications for the representation of metadata can be handled by RDF at this level. Examples include card index information (the Dublin Core), Privacy information (P3P), associations of style sheets with documents, intellectual property rights labeling and PICS labels. We are talking about the representation of data here, which is typically simple: not languages for expressing queries or inference rules.

RDF documents at this level do not have great power, and sometimes it is less than evident why one should bother to map an application in RDF. The answer is that we expect this data, while limited and simple within an application, to be combined, later, with data from other applications into a Web. Applications which run over the whole web must be able to use a common framework for combining information from all these applications. For example, access control logic may use a combination of privacy and group membership and data type information to actually allow or deny access. Queries may later allow powerful logical expressions referring to data from domains in which, individually, the data representation language is not very expressive. The purpose of this document is partly to show the plan by which this might happen.

The Schema layer

The basic model of the RDF allows us to do a lot on the blackboard, but does not give us many tools. It gives us a model of assertions and quotations on which we can map the data in any new format.

We next need a schema layer to declare the existence of new property. We need at the same time to say a little more about it. We want to be able to constrain the way it used. Typically we want to constrain the types of object it can apply to. These meta-assertions make it possible to do rudimentary checks on a document. Much as in SGML the "DTD" allows one to check whether elements have been used in appropriate positions, so in RDF a schema will allow us to check that, for example, a driver's license has the name of a person, and not a model of car, as its "name".

It is not clear to me exactly what primitives have to be introduced, and whether much useful language can be defined at this level without also defining the next level. There is currently a RDF Schema working group in this area. The schema language typically makes simple assertions about permitted combinations. If the SGML DTD is used as a model, the schema can be in a language of very limited power. The constraints expressed in the schema language are easily expanded into a more powerful logical layer expressions (the next layer), but one chose at this point, in order to limit the power, not to do that. For example: one can say in a schema that a property foo is unique. Expanded, that is that for any x, if y is the foo of x, and z is the foo of x, then y equals z. This uses logical expressions which are not available at this level, but that is OK so long as the schema language is, for the moment, going to be handled by specialized schema engines only, not by a general reasoning engine.

When we do this sort of thing with a language - and I think it will be very common - we must be careful that the language is still well defined logically. Later on, we may want to make inferences which can only be made by understanding the semantics of the schema language in logical terms, and combining it with other logical information.

Conversion language

A requirement of namespaces work for evolvability is that one must, with knowledge of common RDF at some level, be able to follow rules for converting a document in one RDF schema into another one (which presumably one has an innate understanding of how to process).

By the principle of least power, this language can in fact be made to have implication (inference rules) without having negation. (This might seem a fine point to make, when in fact one can easily write a rule which defines inference from a statement A of another statement B which actually happens to be false, even though the language has no way of actually stating "False". However, still formally the language does not have the power needed to write a paradox, which comforts some people. In the following, though, as the language gets more expressive, we rely not on an inherent ability to make paradoxical statements, but on applications specifically limiting the expressive power of particular documents. Schemas provide a convenient place to describe those restrictions.)

A simple example of the application of this layer is when two databases, constructed independently and then put on the web, are linked by semantic links which allow queries on one to converted into queries on another. Here, someone noticed that "where" in the friends table and "zip" in a places table mean the same thing. Someone else documented that "zip" in the places table meant the same things as "zip" in the employees table, and so on as shown by arrows. Given this information, a search for any employee called Fred with zip 02139 can be widened from employees to in include friends. All that is needed some RDF "equivalent" property.

The logical layer

The next layer, then is the logical layer. We need ways of writing logic into documents to allow such things as, for example, rules the deduction of one type of document from a document of another type; the checking of a document against a set of rules of self-consistency; and the resolution of a query by conversion from terms unknown into terms known. Given that we have quotation in the language already, the next layer is predicate logic (not, and, etc) and the next layer quantification (for all x, y(x)).

The applications of RDF at this level are basically limited only by the imagination. A simple example of the application of this layer is when two databases, constructed independently and then put on the web, are linked by semantic links which allow queries on one to converted into queries on another. Many things which may have seemed to have needed a new language become suddenly simply a question of writing down the right RDF. Once you have a language which has the great power of predicate calculus with quotation, then when defining a new language for a specific application, two things are required:

• One must settle on the (limited) power of the reasoning engine which the receiver must have, and define a subset of full RDF which will be expected to be understood;
• One will probably want to define some abbreviated functions to efficiently transmit expressions within the set of documents within the constrained language.

See also, if unconvinced:

• What the Semantic Web is not - answering some FAQs

The metro map below shows a key loop in the semantic web. The Web part, on the left, shows how a URI is, using HTTP, turned into a representation of a document as a string of bits with some MIME type. It is then parsed into XML and then into RDF, to produce an RDF graph or, at the logic level, a logical formula. On the right hand side, the Semantic part, shows how the RDF graph contains a reference to the URI. It is the trust from the key, combined with the meaning of the statements contained in the document, which may cause a Semantic Web engine to dereference another URI.

Proof Validation - a language for proof

The RDF model does not say anything about the form of reasoning engine, and it is obviously an open question, as there is no definitively perfect algorithm for answering questions - or, basically, finding proofs. At this stage in the development of the Semantic Web, though, we do not tackle that problem. Most applications construction of a proof is done according to some fairly constrained rules, and all that the other party has to do is validate a general proof. This is trivial.

For example, when someone is granted access to a web site, they can be given a document which explains to the web server why they should have access. The proof will be a chain [well, DAG] of assertions and reasoning rules with pointers to all the supporting material.

The same will be true of transactions involving privacy, and most of electronic commerce. The documents sent across the net will be written in a complete language. However, they will be constrained so that, if queries, the results will be computable, and in most cases they will be proofs. The HTTP "GET" will contain a proof that the client has a right to the response. the response will be a proof that the response is in deed what was asked for.

Evolution rules Language

RDF at the logical level already has the power to express inference rules. For example, you should be able to say such things as "If the zipcode of the organization of x is y then the work-zipcode of x is y". As noted above, just scattering the Web with such remarks will in the end be very interesting, but in the short term won't produce repeatable results unless we restrict the expressiveness of documents to solve particular application problems.

Two fundamental functions we require RDF engines to be able to do are

1. for a version n implementation to be able to read enough RDF schema to be able to deduce how to read a version n+1 document;
2. for a type A application developed quite independently of a type B application which has the same or similar function to be able to read and process enough schema information to be able to process data from the type B application.

(See evolvability article)

The RDF logic level is sufficient to be usable as a language for making inference rules. Note it does not address the heuristics of any particular reasoning engine, which which is an open field made all the more open and fruitful by the Semantic Web. In other words, RDF will allow you to write rules but won't tell anyone at this stage in which order to apply them.

Where for example a library of congress schema talks of an "author", and a British Library talks of a "creator", a small bit of RDF would be able to say that for any person x and any resource y, if x is the (LoC) author of y, then x is the (BL) creator of y. This is the sort of rule which solves the evolvability problems. Where would a processor find it? In the case of a program which finds a version 2 document and wants to find the rules to convert it into a version 1 document, then the version 2 schema would naturally contain or point to the rules. In the case of retrospective documentation of the relationship between two independently invented schemas, then of course pointers to the rules could be added to either schema, but if that is not (socially) practical, then we have another example of the the annotation problem. This can be solved by third party indexes which can be searched for connections between two schemata. In practice of course search engines provide this function very effectively - you would just have to ask a search engine for all references to one schema and check the results for rules which like the two.

Query languages

One is a query language. A query can be thought of as an assertion about the result to be returned. Fundamentally, RDF at the logical level is sufficient to represent this in any case. However, in practice a query engine has specific algorithms and indexes available with which to work, and can therefore answer specific sorts of query.

It may of course in practice to develop a vocabulary which helps in either of two ways:

1. It allows common powerful query types to be expressed succinctly with fewer pages of mathematics, or
2. It allows certain constrained queries to be expressed, which are interesting because they have certain computability properties.

SQL is an example of a language which does both.

It is clearly important that the query language be defined in terms of RDF logic. For example, to query a server for the author of a resource, one would ask for an assertion of the form "x is the author of p1" for some x. To ask for a definitive list of all authors, one would ask for a set of authors such that any author was in the set and everyone in the set was an author. And so on.

In practice, the diversity of algorithms in search engines on the web, and proof-finding algorithms in pre-web logical systems suggests that there will in a semantic web be many forms of agent able to provide answers to different forms of query.

One useful step the specification of specific query engines for for example searches to a finite level of depth in a specified subset of the Web (such as a web site). Of course there could be several alternatives for different occasions.

Another metastep is the specification of a query engine description language -- basically a specification of the sort of query the engine can return in a general way. This would open the door to agents chaining together searches and inference across many intermediate engines.

Digital Signature

Public key cryptography is a remarkable technology which completely changes what is possible. While one can add a digital signature block as decoration on an existing document, attempts to add the logic of trust as icing on the cake of a reasoning system have to date been restricted to systems limited in their generality. For reasoning to be able to take trust into account, the common logical model requires extension to include the keys with which assertions have been signed.

Like all logic, the basis of this, may not seem appealing at first until one has seen what can be built on top. This basis is the introduction of keys as first class objects (where the URI can be the literal value of a public key), and a the introduction of general reasoning about assertions attributable to keys.

In an implementation, this means that reasoning engine will have to be tied to the signature verification system . Documents will be parsed not just into trees of assertions, but into into trees of assertions about who has signed what assertions. Proof validation will, for inference rules, check the logic, but for assertions that a document has been signed, check the signature.

The result will be a system which can express and reason about relationships across the whole range of public-key based security and trust systems.

Digital signature becomes interesting when RDF is developed to the level that a proof language exists. However, it can be developed in parallel with RDF for the most part.

In the W3C, input to the digital signature work comes from many directions, including experience with DSig1.0 signed "pics" labels, and various submissions for digitally signed documents.

Indexes of terms

Given a worldwide semantic web of assertions, the search engine technology currently (1998) applied to HTML pages will presumably translate directly into indexes not of words, but of RDF objects. This itself will allow much more efficient searching of the Web as though it were one giant database, rather than one giant book.

The Version A to Version B translation requirement has now been met, and so when two databases exist as for example large arrays of (probably virtual) RDF files, then even though the initial schemas may not have been the same, a retrospective documentation of their equivalence would allow a search engine to satisfy queries by searching across both databases.

Engines of the Future

While search engines which index HTML pages find many answers to searches and cover a huge part of the Web, then return many inappropriate answers. There is no notion of "correctness" to such searches. By contrast, logical engines have typically been able to restrict their output to that which is provably correct answer, but have suffered from the inability to rummage through the mass of intertwined data to construct valid answers. The combinatorial explosion of possibilities to be traced has been quite intractable.

However, the scale upon which search engines have been successful may force us to reexamine our assumptions here. If an engine of the future combines a reasoning engine with a search engine, it may be able to get the best of both worlds, and actually be able to construct proofs in a certain number of cases of very real impact. It will be able to reach out to indexes which contain very complete lists of all occurrences of a given term, and then use logic to weed out all but those which can be of use in solving the given problem.

So while nothing will make the combinatorial explosion go away, many real life problems can be solved using just a few (say two) steps of inference out on the wild web, the rest of the reasoning being in a realm in which proofs are give, or there are constrains and well understood computable algorithms. I also expect a string commercial incentive to develop engines and algorithms which will efficiently tackle specific types of problem. This may involve making caches of intermediate results much analogous to the search engines' indexes of today.

Though there will still not be a machine which can guarantee to answer arbitrary questions, the power to answer real questions which are the stuff of our daily lives and especially of commerce may be quite remarkable.

One way of looking at the Semantic Web is a breaking down of barriers between applications. An example I have often quoted is that I would like to be able to drag a photo album onto a calenar application and see the photos on my calendar.

What would this actually take?

The intersting case that the calendar application and the phto album application were written indepedently, without this case in mind. Suppose they were, though written to be semantic-web aware. So they will have ontologies for the things they deal with (photos, events resepctively in this case.) They will store their data, or at least a copy of it, in RDF.

The clipboard, which stores things which we copy and paste, is a complex thing. It doesn't always hold the value of something clipped, it sometimes just remembers where it was clipped from. There is a form of negotiation between the source and destination about the format for the data to be transferred. This is why you can copy from text from a web page (which is structured hypertext) into a plain text message. There is a negotiation, and the source and destinatio can both use a plain text clipboad type.

In this example, we can imagine there being a Semantic Web clipboard type. The data is basically transferred as an RDF graph. But that isn't the end of it. Different applications understand different vocabularies (ontologies). So a semantic web clipboard (or the application) does more than just transfer data. It arranges to convert it into a useful form.

When you drag something onto the calendar application, it may be expecting events. It may be able to use anything which has at least a start datetime and and a description. the typical vocabulary here is iCalendar-like, such as @@@@. The photo has a date of creation and may have a form of description. It will typically have technical details of the exposure. The typical vocabulary here is EXIF-like, such as @@@@.

RDF Interest group people have looked at conversion tools. At MIT we've had fun converting things like this deliberately. How can it happen in this example?

Somehow, the user must authorize conversion rules to be available to the semantic web clipboard, so that the convertion can be done. The clipboard indexes the rules, knows what form of information is needed by the application through some kind of registration, and knows what sort of information is available on the clipboard.

Typically, these things are customized. I might like the description of the event of a phtograph being taken to have a list of the people in the shot, if it is known; others might just want the brief "Pic!".

Sources and organization of rules

An advanced user who uses some kind of tool to generate a rule, as users do today for email filtering. The user is a third party. Third party rule sources may include system administrators.

The source could be the creator of one or other program: calendar or photo album. One might exect the program which is released second to come with rules for connection to other things already released. In this case, there is some order in that the rules link applictions in a directed way. If in laterreleases both applications offer rules to convert the same way, then a choice has to be made, just as one sometiems has to chose whether to trust the provider of the printer or the provider of the operating system when installing a printer driver.

Users will have to track the trust worthiness of many different sources of data, but these rules will give a lot back for something quite small.

There is a form of entropy which increases as these rules are used. Some rules may be reversable, but typically they are not. You can't turn every event back into a picture, and you can't event turn a picture taking event back into the picture as you have lost some data. An event is in this example a more generic thing than a picture. One might assume that the system will in general try to reduce this information loss. This will involve trading at the highest level, or usingthe fewest rules. The sense of specificity might therefore form an organizing technique. This is similar to a form of pecking order between a rich text clipboard and a plain text clipboard: is applications can use the more sophisticated, less information is lost.

Conclusion

The Semantic Web clipboard might be a nifty hack in the short term, might be a mainstay of desktop interoperability in the future. From the reseacrh point of view, the rule management involved is a miniature version of the Semantic Web rule indexing search engine.

Update

See the 2009 DIG Semantic Clipboard project by Oshani Seneviratne.

Read whole article...

This note is a little motherhood and apple pie about how a specification should be couched so as to clearly add a new well-defined piece to the technology.

A technical specification defines something. The document must specify the thing being defined as well as give its definition: a "left hand side" as well as a "right hand side". Both must be done quite precisely.

Typically, technical specifications for the web specify a language or a protocol. A protocol is a language for messages, plus a set of constraints on the sequence of messages. A language is a set of symbols, the syntactic constraints on the way their are combined, and the semantics of what they "mean" at some (possibly more than one) level. (See also Meaning of web documents)

The test of a good specification is that it clearly defines what implementation (document, message, program) conforms, and of course that it ensures by its design that whatever conforms works to provides the required function.

The left hand side

The document should state what sort of things is being defined. It should introduce a new term which characterizes that which conforms to the specification. Examples of a conformance term could be

• A well-formed XML 1.0 document
• A conforming HTTP 1.1 server
• An xml-schema-valid XML document
• A W3C/WAI "AAA" accessible web site
• The SVG 1.0 language

The same specification document can define more than one term. such as a "strictly conforming WWidget" and a "loosely conforming WWidget" but one should beware of diluting the "WWidget" brand.

As systems become more self-describing, the term is given a formal identifier. Examples could be

• The MIME type "image/svg1"
• The XML Namespace "http://www.w3.org/1999/asdf-2-0"

In this case where a MIME type or namespace has an identifier, then this is obviously the crucial term to use to be unambiguous.

Wherever possible, conformance phrases will be grounded in the Web: identified by a URI.

The Right Hand Side

More has already been written on this, and most of it seems to be in consensus in the W3C.

It is important to remember what you are defining as you write the text. For example, if you are defining a "foo-valid document" then using "is invalid" in the text can be assumed to apply to this but "is incorrect" or "is wrong" or "produces an error" does not unless the language is explicitly linked to the conformance term.

A good spec similarly pays attention to:

• The distinction between the use of MUST, as opposed to MAY (etc., see Bradner's BCP14)
• The use of this distinction in defining the conformance term precisely;
• The distinction between normative and non-normative parts of the specification.

When defining a language, whenever possible specify directly the meaning rather than the sort of thing you would expect some software to do with it. Typical behaviors of an agent may be very useful to explain the intent non-normatively.

For example,

"x" indicated that the check is void

is better than

"x" indicates that the check should be rejected with a fatal error.

You can tell people what something means, you can't tell them what to do about it, unless you are defining a protocol. When defining a protocol, then the constraints should ideally be given as a state transition diagram or table to make them totally clear.

When defining a message which in fact binds to human social entities, then this must be clear. You could end up in court explaining it if not. ("The MMTP protocol defines the meaning of a message sent by or on behalf of a party herein referred to as the "debtor" to a party referred to as the "creditor". The creditor is identified by the foo-email-address...)

When defining a part of a specification deliberately to be similar to another specification,

• Make it clear that you have noticed the similarity;
• Make it clear whether the similarity is exact and if not where not (and why not);
• Ideally, it clear that the existing specification is being referred to normatively and is definitive, and that what is in this specification is a non-normative copy for information only.
• Make it clear whether the use in this specification will track any new version of the referenced specification.
• Think about whether there is any way in which such changes could break this system.
• If necessary negotiate constraints with the authority for changes to the referenced specification.

Test questions

A few examples of things to ask about a spec -- though generalization is difficult.

• Does the spec give enough information to determine, for any arbitrary object, whether the conformance term applies to it?
• Could you write a program to test conformance?
• Is conformance alone enough to ensure that systems build using this language will function as intended and with integrity?
• Can you prove important properties of the system from the state transition tables etc?

So much for another bit of folklore. Comments, suggestions welcome.

References

S. Bradner, "Key words for use in RFCs to Indicate Requirement Levels", BCP0041

Bits mean something.

When you connect a cat-5 ethernet cable to your computer, you effectively commit to taking part, with your computer, in a very special system. It is a system in which the meaning of messages is determined, in advance, by specifications. This is a principle which is so basic to network computer systems that it is rarely stated. But as the stack of specifications gets higher and higher, and as electronic commerce, legally enforceable agreements, and socially sensitive issues such as privacy and fraud become matters of public concern, it is worth reiterating for the record.

The Internet works because of interoperability between different computers, despite different hardware, operating systems, local language context, and software supplier. Users of the web sign on to the use of these languages when they use the Internet.

There is this little philosophy joining many specifications, without which the Web falls apart.

Lets take an example.

You have an ethernet cable

You walk into a meeting room, and you are offered a thin cat-5 cable with a 10-base T connector. This is an Ethernet connector which only takes Ethernet packets. The only way to use it to communicate is for your computer to send packets which are formatted to the Ethernet specification. The Ethernet specification is a large document (Similar to IEEE standard 802.3) put together by a bunch of engineers, and once they were done Ethernet existed as a standard, and computers which know nothing about each other could exchange packets over local area networks..

The Ethernet defines the format of an Ethernet packet, which has a little header information, but mostly carries information on behalf you the user. The spec also, importantly, defines some rules of behaviour. For example, the ethernet doesn't work if more than one computer tries to transmit at once. There is a rule that if you find that happens, everyone involved backs off and comes back at a random interval. Each computer is supposed to wait on average the same amount of time before trying again. Of course, you could cheat by actually pretending that your random number happened to be really small every time, and on average your computer would end up getting though more and blocking everyone else out, just like people who always seem to be the one talking in a meeting. But that would be cheating, and contrary to the Ethernet specification. By connecting to an ethernet cable, there is an understanding that your computer will stick to the rules

An ethernet packet can be sent to anyone on the same wired or wireless local area network. How does a computer know what to do with a packet when it gets it? How does it know how to interpret that packet? Well, there is a field in the packet which tells it, in a coded way, what the use of the packet is, and therefore how to interpret it.

Of course, there are lots of uses of the Ethernet, but a very common use of an Ethernet packet is to use it to carry an Internet packet. Ethernet packets can only cross the local area network, while Internet packets are forwarded anywhere in the world. So, there is a particular code - a particular value for the field in the Ethernet packet - which tells any receiving computer that the data is actually an Internet Packet. This means that to understand anything more about the packet means, you have to read another spec: the Internet Protocol (IP, RFC791).

@@@ The complete graph of interdependencies between specifications.

You send an Internet packet

So suppose you send an Internet packet. You put the ethernet address of the local "router" computer into the ethernet address field, but within the "data" part of the ethernet packet is the IP packet and inside that is an internet address field, which takes the IP address (the thing like 18.96.237.175) which identifies the computer Although the ethernet packet you send it in only gets as far as some computer a "router" on the local net, that computer passes the IP contents on, from computer to computer across interconnected networks until it arrives on the right local network for its actual destination.

So how does that computer know what to do with it? Well, there is a field in the IP packet which carries a coded value to tell the computer receiving it what to do with it. .

From Internet Protocol (RFC791):
A summary of the contents of the internet header follows:

      0                   1                   2                   3
      0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |Version|  IHL  |Type of Service|          Total Length         |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |         Identification        |Flags|      Fragment Offset    |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |  Time to Live |    Protocol   |         Header Checksum       |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |                       Source Address                          |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |                    Destination Address                        |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |                    Options                    |    Padding    |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
                      Example Internet Datagram Header

                                 Figure 4.

And there are a lot of things you can do with an IP packet, but a very common one is to use that IP packet to set up, or to be a part of, a reliable stream of communication using the Transmission Control Protocol (TCP) (RFC 793).

You send a TCP packet

When you send, or your computer sends, a packet in the TCP protocol, there is an understanding that that packet conforms to the protocol. That means a couple of things. It means that you agree that the packet's contents it to be interpreted according to the TCP protocol specification. It also means that you agree to abide by the rules of the specification, which determine, rather like with the Ethernet protocol, how long your computer will wait before re-sending a packet which didn't seem to get there. If your computer re-sends too early, then it hogs the Internet and slows down everyone else. If your computer send a packet to start a new connection when it doesn't really want to, then the destination computer will prepare a lot of memory to receive all the data you are going to send, and wait. If you keep doing it, then that computer can just run out of memory and stop working. So you can cheat and you can do real damage by breaking protocols.

Introducing IANA: The Port number registry

So you computer must stick to the TCP specification. When it does that, the TCP protocol assures that the two computers have a reliable connection without any missing bits. What they use it for is no concern of TCP, apart from the fact that the TCP protocol specifies, within the TCP packet (which is inside the IP packet (inside the ethernet packet)) a special field whose coded value, or port number. There is a convention, which is written into the TCP specification, (@@check and quote wording) that the meaning of the port number is determined by a table which is changed from time to time, but kept by the Internet Assigned Numbers Authority (IANA). Without going into the politics of the changes and control around IANA, it is just worth noting that this is, architecturally, a "flexibility point", where the community can introduce a new protocol to run on top of TCP/IP without having to write it into a new version of the TCP/IP specification itself.

The port number registry is on the web (@@ link) but also, on a unix computer, there is a list of the well-known ports in the file /etc/services.

When you send a TCP/IP packet there is therefore an understanding that if you send to one of the well-defined port numbers, then you are going to use it in a way defined by the specification defined in the IANA registry. For example, port number 25 indicates that you are going to use it to transfer some email, and that you undertake to communicate according to the Simple Mail Transport Protocol specification.

You send an email message

You get the picture. One specification, once you commit to it, depending on the values of certain fields, invokes further specifications. By committing originally to using an ethernet cable, you commit to your computer using, on your behalf, the various other specifications. In the case in which your computer sends email, it may for example open a TCP/IP connection to to port 25, and then use the Simple Mail Transfer Protocol (SMTP, RFC821). This specification indicates that the body of the SMTP communication is formatted according to the email message specification, RFC822. RFC 822 specifies the headers on email messages. It specifies, for example that a given "From" field indicates the email address sender of the message.

It is possible, of course, to cheat. with the SMTP protocol. It is possible to lie about who is sending the message - to send a message which appears to be from one person to a friend. This breaks to protocol. It breaks it, here, in a way which is very clear to people: it sneaks past their personal email filtering, and also any automated filtering, tricking them into reading a message. This is a security violation. It can use up a person's time, energy, bandwidth and disk space for the commercial gain (indirectly through advertising and sales) of the perpetrator.

The Internet specifications, to which any Internet user implicitly agrees in using the Internet at all, define what the fields in an email message mean. To put incorrect information in these fields is to make a misrepresentation, just as it would have been in any other medium. It should be subject to the same penalties as lying or fraud in any other medium.

When the Internet was young and used by research institutions, its misuse would inconvenience other users and lead to reprobation and the disdain of one's peers. Now that the Internet is such as large force in society, it is possible to make a lot of money and create a lot of damage by protocol abuse. You can compare a lie in an internet message, depending on how it is done, to forging a check, connecting to the electricity supply the other side of the meter, or to poisoning the water supply. Society must therefore be careful to be absolutely clear about the illegality of such misuse.

You publish a Web page

When you publish a web page, just as when you send an email message, the web page or the message generally carries a meaning. Well,it can be a picture or a poem which is more artistic than linguistic, but in a large number of cases the meaning is a well-defined part of a communication between parties. It may be a human-readable document, like the page describing a pair of pants your are about to buy from a store, or it may be machine-processable, like the Online Financial Exchange (OFX) format bank statement your financial software downloads from your bank.

Of course, you would find it hard work to make sense of the OFX file if you just read it without the help of the financial agent, and your financial agent wouldn't make much sense of the catalog page. Something must allow us to distinguish how web pages and emails should be interpreted, just as a computer has to figure out how to make sense of an Ethernet packet. And just the same sort of thing indeed happens.

When you publish a web page, you give it a HTTP URI. You pick a URI from the space of URIs which are yours to define. Some people have space on their own domain, some people have the right to pick URIs in part of someone else's domain. But the URI is one which you own or over which you have authority. You are not allowed to pick one in someone else's space.

Whoever owns the domain has the authority to define which computer serves information in it. They have the authority then to have a computer -- a web server - which is configured to act on their behalf. It is then assumed that the computer acts on the their behalf. The server is the agent of the publisher. What it does is tell any asking browser what you have said is a representation of the document for a given URI.

When someone follows a link to your web page, their browser opens a TCP/IP connection to TCP port 80 on the machine which is registered as serving the (www.whatever.com, etc) in question. Their agent, their browser, asks your agent, the server, to give it some representation of the web page for that URI.

Why? Because the URI specification says that what you can tell about a URI depends on the first bit, in this case http:. It indicates that an IANA URI scheme registry is used to tell you what specification applies.

The IANA registry indicates that the http: scheme calls out the HTTP 1./1 specification, RFC@@@.

HTTP 1.1 says that (unless otherwise specified) the client contacts the server on TCP port 80. The IANA registry of port numbers, just as it allocates port 25 to mail transfer, allocates 80 to HTTP. The HTTP spec is therefore mutually assumed by both parties. This spec describes what a request means, and that when the request is successful, what the response message sent back to the browser means.

According to HTTP 1.1, in that response, there is a field (Content-type) which indicates how the body of the response should be interpreted. For each valid value of that field, there is an IANA content-type registry value which explains which specification applies to the body of the message. This is just the same system as for email.

When the value if the field is text/html, it indicates that the message is a hypertext document ("web page") which is to be presented to the human being and interpreted then by the human being in the usual human way. If the field indicates it is an OFX file, then that means that the OFX specification determines what it means, and you need a program or something which understands what the fields of the OFX documents mean. In neither case can you argue that you didn't know. So long as the writers of the specification do a good job (and goodness knows they work hard enough at it) then there can be no argument as to what the actual fields in your bank statement mean.

You publish an XML document

When you publish a document in XML, then there is another layer involved. Many different languages -- or even mixture of languages -- can be sent structured as XML. The mime type of the document can just be "application/xml", which doesn't tell the reader how to interpret it. For that, you have to look at the outermost element of the XML document. The namespace declaration gives a URI indicating the namespace.

Note the difference between the use of a URI and a central registry. Because the namespace is identified by a URI, the web becomes the registry. Anyone can make a new XML namespace. Also, one can use a URI, such as a HTTP URI, which can be dereferenced. This allows the information which would have been in the registry to be put into a web document. (The W3C TAG is currently debating the issue of the best format to use for this meta information, but HTML, RDDL and RDF have been used in various combinations. But broadly there are two types of information. There may be a specification (or a reference to one) to tell a human reader what the language is and how to interpret it. there may also be data - a schema which describes the grammar of the language, or even the start of a logical definition of what the language means.

But whatever information may or may not be available automatically, in an XML world, a system has to look into the document, at the namespace of the outermost element, to know how to interpret it. This generally means what application to launch - not to mention what icon to use to represent the document to a person.

An example of a machine-readable document with important semantics is an online P3P web site privacy policy. This is an XML document which gives, for each category of personal information, the sort of thing the web site promises to do or not do with it. It can be scanned by a a browser more easily than a person can read a privacy policy. It is a useful feature, as it saves everyone's time and increases public confidence in responsible web sites. It clearly depends on the meaning of the terms being well defined by the specification.

(Problem: this doesn't always happen: MathML and XHTML as XML in practice.@@ links)

You publish an RDF document

Now let's talk semantics. Harder semantics - for logical systems. Some XML documents are RDF documents. RDF/XML is an XML-based language for data. It is very simple: each document is just a set of "triples". A triple gives the value of some property of some object - or some relationship between some object and some other object. The triples are independent, so interpreting the document is just, the RDF spec explains, a question of interpreting each triple.

How do you figure out what a triple means? Well, the property (or relationship) is identified by a URI. And whoever made up the URI gets to say what the property means, that is, what any triple using that property means.

So if make a property http://www.w3.org/2002/05/example#color and define that the color property is a name out of the Pantone(tm) list of colors and you send someone an order in RDF for a hat which has http://www.w3.org/2002/05/example#color of blue256 then you are specifying blue256 on the pantone scale. No one can argue that you meant some other scale of blue. Normally the argument is made much easier by my actually writing a document http://www.w3.org/2002/05/example in which I explain what #color means. No one can argue, in their catalog, that "By suit, we mean something which is black, whatever http://www.w3.org/2002/05/example#color someone might say it is". The meaning of the triple is determined by the property, not by the subject or the object of the triple.

Looking at the table which summarizes the steps we have been through, you will see the specs are connected by some field which points to the next spec through some list or registry. For the more recent layers, the registry has been replaced by the Web.

The hooks - identifiers

That's an interesting trend. If you like, we can see the technology move through three stages of civilization, in terms of the identifiers which are used for concepts.

1. Using numbers or strings
2. Using URIs - identify the same thing in all contexts
3. Using dereferencable URIs

The early protocols used numbers and strings which requires a central registry. that worked, because the only common concepts were those in the standard protocols, and those had to be common across the net for interoperability. In these areas still there is a strong argument for central control.

As we move on to later protocols, the protocols themselves become more diverse. This is partly because they are at a higher application level. The centralized model starts to break down, as witness some of the social difficulties of getting an IANA allocation for a MIME type an embryonic W3C specification. So new protocols allow new applications to be defined using URIs, allowing anyone who has access to a bit of domain space to allocate them.

The third stage of civilization is the one at which the identifiers can be looked up on the web. This is quite useful for engineers who encounter new languages. It doesn't really justify its existence, though, until one has technology -- Semantic Web technology -- in which an automated agent can pick up metadata about the languages on the fly, and use that metadata to enhance its processing of data in that language.

(What if I don't have a web site? This is becoming less and less of a problem. There are all kinds of existing ways of allocating an identifier. But the persistence of such information is, and always will be, like the cleanliness of water and air, an important social issue.)

When the chain does NOT connect

We have seen how any user of the Internet is bound to a series of specifications which define the meanings of terms, and hence allow his or her equipment and agents to interoperable with others. This stack prevents one from sending a nasty email to someone and then protesting that the message didn't mean anything. So if the stack is so strict, how does one send a nasty email message when one doesn't mean it? There are plenty of times you want to include an attachment to which you want to refer, but for which you don't claim authorship or responsibility. Understanding the exceptions is as important as understanding the general rule. Many protocols have ways of breaking the chain, of including information which is not part of the meaning of the message.

In email it is an attachment. There is always in email a cover note, the basic message, which conveys the actual message. You normally only use any attachment according to the main message. It might be "Hey, Joe, what do you think of this paper?", or "Look at this stupid program - but whatever you do don't run it!"

Currently (2002) XML doesn't have a common standard for what has been called in that context "packages". This is a pity. It is on the agenda for XML Protocol working group, as seen as essential for SOAP operations. One must be able to include documents stapled to a SOAP request or response, which are not to be just acted on.

At the Semantic Web level, those who have played with the Notation3 language will recognize the curly brackets as the packaging, or quoting. Whereas a document

my:car  srgb:color "000044".

asserts that the car in question is blue, the document

my;form67  :says {my:car  srgb:color "000044"}.

does not. It merely says something about the statement that the car is blue.

So being able to refer to something without asserting it, whether you call it attachment, packaging, or quoting, is an important feature of a language. The fact that you can do this removes the last excuse for anyone claiming not to have meant whatever they did say in the main message!

Conclusion

Internet messages and Web documents are represented in computer languages with well-defined specifications. Use of the Internet and the Web implies an acceptance of the specifications as authoritative.

The specifications are linked together by identifiers which in earlier specs were numbers, but in later specs are URIs, ideally URIs which can be looked up on the Web. The ability to make these linked specifications requires the specifications to be designed very independently. This is simply the software engineering practice of information hiding between layers.

The trend for the higher layers is toward more and more machine-processable metadata about such languages, which can be retrieved automatically and will aid in processing. Some of these will relate the semantics of terms in one vocabulary to terms in another, on a web-like way.

The fact that as we move into the applications we see more and more diverse uses of the Web and the Net does not diminish our reliance on a sound standards in the supporting infrastructure.

We celebrate the web as a space which allows you to discover all kinds of new things people and places. We admire it as something which breaks down geographical barriers, allowing me to interact with someone on the other side of the world as easily as with the person next door. But are we really using that aspect, of it?

When we log on to a socials networking site, the site keeps a record of the people we now and suggest new people which we should know or maybe do know. The simple way xdoing this is just to pick people who are friend s of our existing friends. We are almost sure to like them, especialy if we have a lot of friends in common. So what happens if we accept these new proposed friends? Out social network becomes a solid mass of mutually connected people. We make cliques. What's wrong with this picture? It doesn't in fact add new social experience. In fact, as Eli Pariser describes in "The Filter Bubble" and his TED talk, we in fact can end up constructing a world in which we only meet people who act and think like us. So it for example, that when kids have the ability on the Net to play a multiplayer online game with anyone logged on in the whole world, they end up playing with the kid next door, maybe even the kid at the other end of the couch. This is good for the cohesion of the neighborhood, but is it good for the planet?

Supposed once in a while when the system suggests a friend, it offers a Stretch Friend. A stretch friend may be almost Identical to you, but differs along one axis. You are a white male liberal geek in New York and you are offered a black male liberal geek in New York or maybe a white male liberal geek in Saudi Arabia. It get the idea... You are asked to cross one boundary, of race, color, creed, intellectual interests, political leaning, culture, location. Just one at a time. Maybe more later, but for now, let's see how you get on. Can you put the effort in to talk to someone whose life or beliefs or interests differ along one axis?

If you could, and lots of people had stretch friends, would the world be an incrementaly better place, or maybe after a while a dramatically different place?

Nate Silver, who is famous for his insight into how people vote, in a TED talk @@@ explains his puzzlement that why in the election which first elected Barack Obama to the US presidency, thee was a general shift toward the liberal side across the whole country ... Except for a bit of it. Certain demographic didn't shift, and he wondered why. Who were they? He found they were the people who, when asked, said they just could not imagine a black person being president if the USA. And who were those people, then? Well, they were by strong correlation the people who answered "no" to the question, "Have you ever lived or worked with someone of a different race". @@check details

So human beings have an ability to mark people outside their normal zone as being strange, or foreign, or enemy. We have evolved the neurology to become tribal at the drop of a hat, as that rather horrid experiments showed when school children wore badges saying whether they had blue or brown eyes.b@@ ref. Breaking down these boundaries, these differences of a single quality or situation which can otherwise allow us to switch into the tribal mode, therefore becomes an matter of urgency.

The stretch friend idea can obviously be done in a number of ways and expanded into more complicated schemes. It could be connected with twinning projects in which whole towns or schools pair up with partner towns or schools in very different parts of the world. (Of course when these are arranged by churches, they don't always cross religious boundaries!)

Web scientists, people who analyze these things, could do lots of math to try work out what the effect would be. Or we could just try it.

In 2004 there were proposals to create new top-level domains which included .mobi and .xxx. There are major problems with these proposals. There are costs in general to creating any new top level domain. There are specific ways in which the ".mobi" breaks the Web architecture of links, and attacks the universality of the Web.

At their 14 May 2004 face-to-face meeting, the W3C Technical Architecture Group resolved to support this document, with Norman Walsh abstaining, and Paul Cotton recusing himself.

Introduction

When the Internet was being collaboratively developed by a substantially technical community around a growing but still manageable Internet Engineering Task Force, the Domain Name System (DNS) evolved as a hierarchical solution to the problem of keeping track of which computers had which Internet Protocol (IP) addresses. The tree structure was an improvement over the previous flat space of host names. It reduced the chaos, by allowing new names to be allocated in sub-domains without recourse to a central registration system. Because the frequency of allocation of new names decreased as one ascended the tree toward the root, the actual cost was kept manageable.

As email and World Wide Web (WWW) use blossomed and became increasingly important, domain names crept out of the messages syntax for Internet protocols and crept into daily parlance. It then became valuable to own a short domain name. This turned domain name space into a limited commodity. After some tussles for control (ongoing at the time of writing) and some large amounts of money changing hands in some cases, the system has now settled down to a market-based one in which names can be rented, transfer value can be asked by the old owner of the new owner, and one-time and annual fees are typically payable by any domain to any company managing the higher domain. An anomaly was that unclaimed names were deemed to have no owner and no value, and were allocated in a "first come first served" frenzy in which speculators made great profits and held to ransom those who may have been considered the more logical owner of a name. This anomaly created great instability. It has costs, in that any trademark owner had to beware of parties who would register domains which included their trademark. The Example Manufacturing Company had to ensure that it owned not only example.com which it had used for email and Web site for many years, but also example.net and example.org to avoid unscrupulous competition setting up sites to benefit from Example's excellent reputation. As the business grew, Example had to also acquire example.fr and example.co.uk to ensure that confusion was minimized.

The fact was that the public memory was not for the domain name, but for the brand name which was sandwiched between www and .com. To this extent, in the world of memorable domain names, the hierarchicalization of the domain system had failed to happen. In the public's memory, example was the mark, and the difference between example.com and example.net merely a source of confusion.

As each node in the tree represents a potentially valuable asset, control of any subset of the tree is valuable. Control of the entire tree is managed by ICANN, which is set up to be a non-profit international institution, with the intent that it should as such carry the trust of the entire community in its efforts to manage the system for the common good. Control of subtrees such as .net, .com and .org is delegated to set of parallel registries whose business model is nominally the charging of registration and annual fees. There have been temptations for the registry companies to consider themselves owners of unclaimed names. Rumors have abounded about systems which would automatically rent a domain name about which a potential renter was inquiring, or would redirect traffic from an unclaimed Web site to their own Web site, and so on.

The Cost of Change

The top level of the domain name system, and to a lesser extent the IP address space, are the single weak, centralised, points of an otherwise strong, decentralised system. The Internet is a net, and the WWW is a Web, but WWW and email use DNS which is a tree, which has a single root. Although there are many benefits to a system with global identifiers, there are also costs, such as a single common DNS tree. As a community we have all decided that the benefits of the system (such as being able to quote example.com anywhere in the world and have it mean the same thing) outweigh the costs of the social systems required to ensure fairness in its operation. There is, however, great stress. ICANN is under constant pressure to alter its balance of power or modus operandi. It balances technical, academic, commercial, and governmental inputs. The whole issue of domain names has created a vast amount of concern. And because the DNS tree is so fundamental to the Internet applications which build on top of it, any uncertainty about the future creates immediately instability and harm.

Our first instincts, then should be not to change the system with anything but incremental and carefully thought-out changes. The addition of new top-levels domains is a very disturbing influence. It carries great cost. It should only be undertaken when there is a very clear benefit to the new domain. In the case of the proposed .mobi domain, the change is actually detrimental.

The Economics of Domain Names

In practice, for most domain name owners, the part between the "www" and the top level domain is their brand, or their name. It is something they need to protect. This means that in practice, a serious organization to avoid confusion has to own its domain in every non-geographical top level domain. For a large company, the cost of this may be insignificant. For a small enterprise, a non-profit organization or a family, the cost becomes very significant.

The chief effect of the introduction of the .biz and .info domains appears to have been a cash influx for the domain name registries. Example Inc. as mentioned above owns example.com, org and .net. Does it also have to buy .biz, .info, and .name to avoid confusion and the misappropriation of my name by others? Will I have to also rent "example.mobi" in case it want to make information available for people who use wireless equipment?

The market for second-level domains is a market for a limited resource. After an unstable period when the first come first served system was in play and greedy squatters grabbed domains simply for speculation, it has now settled down. Introducing new TLDs has two effects.

The first effect is a little like printing more money. The value of one's original registration drops. At the same time, the cost of protecting one's brand goes up (from the cost of three domains to four, five, ...).

The value of each domain name such as example.com also drops because of brand dilution and public confusion. Even though most people largely ignore the last segment of the name, when it is actually used to distinguish between different owners, this increases the mental effort required to remember which company has which top level domain. This makes the whole name space less usable.

Is it fair to reduce the value of these domains which have been acquired at great cost by their owners?

The second effect is that instability is brought on. There is a flurry of activity to reserve domain names, a rush one cannot afford to miss in order to protect one's brand. There is a rash of attempts to steal well-known or valuable domains. The whole process involves a lot of administration, a lot of cost per month, a lot of business for those involved in the domain name business itself, and a negative value to the community.

Fairness

As we have seen, the choice of a tree structure for domain names is one which has costs and benefits, and the community currently accepts both. The cost of confusion, and of extra name registrations, is high. When the benefits of the new domain itself are small or negative (as we discuss below), then one looks for incentive. The large amount of money that has changed hands for domain names might lead a person to suspect that this was the motivation. Under these circumstances, to increase public trust, proposals from non-profit organizations would raise less suspicion.

The root of the domain name system is a single public resource, by design. Its control must be for and, indirectly, by the people as a whole. To give away a large chunk of this to a private group would be simply a betrayal of the public trust put in ICANN.

Specific Problems with .mobi

The different domains are introduced for different reasons, so we must answer this for each one. The ICANN list of proposals gives pointers to the proposals.

The .mobi domain is described as being for the use of a community. To quote the proposal, the target community for the .mobi TLD is:

• Individual and business consumers of mobile devices,services and applications
• Mobile content and service providers
• Mobile operators
• Mobile device manufacturers and vendors
• IT technology and software vendors who serve the mobile community

This is in fact a mixture of reasons. It sounds as though there is a use for ".mobi" when the provider of a service intends it to be for the benefit of mobile users. There appears to be a desire to limit the use of ".mobi" to companies -- perhaps those in the group.

This domain will have a drastically detrimental effect on the Web. By partitioning the HTTP information space into parts designed for access from mobile access and parts designed (presumably) not for such access, an essential property of the Web is destroyed.

Device Independence.

The Web is designed as a universal space. Its universality is its most important facet. I spend many hours giving talks just to emphasize this point. The success of the Web stems from its universality as do most of the architectural constraints.

The Web must operate independently of the hardware, software or network used to access it, of the perceived quality or appropriateness of the information on it, and of the culture, and language, and physical capabilities of those who access it [WTW]. Hardware and network independence in particular have been crucial to the growth of the Web. In the past, network independence has been assured largely by the Internet architecture. The Internet connects all devices without regard to the type or size or band of device, nor with regard to the wireless or wired or optical infrastructure used. This is its great strength. From its inception, the Web built upon this architecture and introduced device independence at the user interface level. By separating the information content from its presentation (as is possible by mixing HTML with CSS, XML with XSL and CSS, etc.) the Web allows the same information to be viewed from computers with all sorts of screen sizes, color depths, and so on. Many of the original Web terminals were character-oriented, and now visually impaired users use text-oriented interfaces to the same information.

For a time, many Web site designers did not see the necessity for such device independence, and indicated that their site was "best viewed using screen set to 800x600". Those Web sites now look terrible on a phone or, for that matter, on a much larger screen. By contrast, many Web sites which use style sheets appropriately can look very good on a very wide range of screen sizes.

It is true that to to optimize the use of any device, an awareness on the part of the server allows it to customize the content and the whole layout of a site. However, the domain name is perhaps the worst possible way of communicating information about the device. Devices vary in many ways, including:

• Network bandwidth at the time,
• Screen size and resolution,
• Intermittent or continuous connectivity,

and so on. While with the current technology, the phrase "Mobile" may equate roughly in many minds to "something like a cell phone", it is naive -- and pessimistic -- to imagine that this one style of device will be the combination that will endure for any length of time. Just as concepts such as the "Network PC" and the "Multimedia PC" which defined profiles of device capability were swept away in the onrush of technology, so will an attempt to divide devices, users and content into two groups. Small devices will have high bandwidth. Devices with large screens will sometimes have small bandwidth. Some "mobile" phones will be permanently mounted on kitchen walls. The range of digital assistants will continue to evolve.

There are good ways to deal with and derive the greatest benefit from the growing diversity of client devices. The adaptation may occur on the client side, the server side, or both. For example, the CC/PP specifications provide a framework for a client device to describe its capabilities in great detail to a server. This is based partly on the UAPROF (User Agent Profile) specifications developed by the mobile phone industry. Also, the HTTP specification has a content negotiation mechanism which allows a device to give a simple profile of its capabilities whenever it asks for some information. Even when a server serves the same static content to mobile and fixed systems, Cascading Style Sheets (CSS) allows specific style information to be applied by hand-held clients only, allowing quite different presentations to be displayed in the two cases. These systems, just a few of the technologies which already exist, leaving aside those which could be designed, are much more powerful than a top level domain name.

The various documents about the ".mobi" Top Level Domain talk about not only mobile devices but "mobile users" and "mobile businesses". There is an indication that the mobile technology providers feel that while one is mobile, or when one is catering to a mobile customer, one is special or different. This may in fact be motivated simply by attempts to increase the visibility of the mobile communications supplier's name. It may be connected with a hope by the communication providers to gain some control of over information flow to and from mobile users. This would be detrimental to the open markets enabled by the Internet.

If neither of these motivations are the cause, then perhaps there is an honest belief that being mobile will indeed be best when it is visible to end users. In other words, the mobile communications providers are expecting to declare failure. It is failure when a communications system, in providing connectivity, becomes foremost in the user's perceptions. A travel agent should be a travel business, not a "mobile business". In a reasonable world, the travel agent gets on with selling flights and not worrying about whether a customer is attached by a wire. In a reasonable world, a phone is a phone and the particular electromagnetics used to connect it to another phone are totally uninteresting compared to the fact that a person is connected to another person.

Damage: Loss of Web Functionality

But the point is not that a division into ".mobi" and the ("immobile?") rest of the world is futile, it is that it is harmful.

The Web works by reference. As an information space, it is defined by the relationship between a URI and what one gets on using that URI. The URI is passed around, written, spoken, buried in links, bookmarked, traded while Instant Messaging and through email. People look up URIs in all sorts of conditions.

It is fundamentally useful to be able to quote the URI for some information and then look up that URI in an entirely different context. For example, I may want to look up a restaurant on my laptop, bookmark it, and then, when I only have my phone, check the bookmark to have a look at the evening menu. Or, my travel agent may send me a pointer to my itinerary for a business trip. I may view the itinerary from my office on a large screen and want to see the map, or I may view it at the airport from my phone when all I want is the gate number.

Dividing the Web into information destined for different devices, or different classes of user, or different classes of information, breaks the Web in a fundamental way.

I urge ICANN not to create the ".mobi" top level domain.

Tim Berners-Lee

Abstract

Existing user interfaces for managing, for example, mail, photos, contacts and songs allow searching using both user-generated 'tags' and also well-defined properties such as the date-time of a photograph, or the values of headers in an email message. Users have on many web sites provided many tags, but their re-use by others has been limited due to the fact that the same tag word has quite different meaning when used by another person or used on another site.

Other data, such 'geotagging' of places, declaration of friends and colleagues, by contrast, have a well-defined meaning and allow query across data from many sites. This article discusses how the user interface metaphor of a luggage label cam used to associate metadata from well-defined ontology with tags from a particular context.

The article discuses ways of encoding the labels in RDF.

Introduction

There are a mixed feelings about the passion for tagging which typifies the Web 2.0 wave. On the one hand, there is excitement about the fact that users are, as a large number, adding re-usable information to the information space, allowing sites such as del.icio.us and flickr to sort, cluster and query masses of otherwise amorphous photos and web content. On the other hand, there is the sinking feeling that tags are headed the same way as keywords of Information Retrieval in the 1980s: initial hope, and then being stranded between the unbearable constraints of a controlled vocabulary and the hopeless ambiguity of uncontrolled user-generated keywords. Tom Gruber, writer of books on ontology who runs a Web 2.0 site himself. gave a talk at ISWC 2006 which touched on bringing the gap, and taking the passion to organize and express, and using it to make re-usable data.

There is currently a tension in the tagging world as to whether tags are regarded as global in meaning, or whether there meaning really depends on the tagger. In del.icio.us, one can query for thinks tagged with a certain word by a certain person. (I heard of one online community which was considering making a system to allow one formally to state when one has committed to use a given tag in the same way as another person, or growing mesh of people. That would be a very interesting feature, as it would allow a useful definition to gain growing acceptance, to progressively move from being a private idea to being a group global standard.)

Meanwhile, other sites get users to provide semantic web data with well-defined global ontologies. The locations of people, events and photos, relationships between people, authorship of publications, things and people an image depicts, and so on, is done using well-defined identifiers (under the covers) for everything involved, including the relationships and properties. The resulting data is extremely re-usable. The problem is that it isn't as quick as tagging with a single word off the top of one's head.

Example: Soccer folders

I face these problems day to day, and like many geeks, am driven by the urge to make the boring things in life happen automatically, with the computer helping more effectively. There are lots of things I can do with N3 rules -- but I'd like to have a nice user interface to it which hides as much technology beneath the surface as possible. I'd like as many non-geeks as possible to be able to use the same tools.

Let's take one example. I took a bunch of photos of a local soccer team, once when they played Wayland, and once when they played Arlington. I loaded them all into iPhoto. I wanted to burn a CD for the team of the best of the bunch. I also want to be able to find them later.

On the first day, I didn't take any other photos, so the simplest thing was to make a 'smart folder' (actually 'smart Album' in iPhoto) , which had in it by definition the photos taken on that day. The smart folder allows you to specify a combination (and or or) a number of constraints such as time, keyword, text and rating. I called this one Soccer vs Wayland.

On the second day, I took other photos as well, so the smart folder was going to be more complicated. So instead, I just found all the photos, selected them, and dumped them in a new plain folder Soccer vs Arlington.

These of course one would represent in RDF as classes. - but we'll get into that later.

Ok, so here's where we get into wish-list territory.

1) At that point, I wanted to be able to make a virtual folder Soccer, and make the two folders subfolders. (There used to be a photo processing tool called Retriever which would handle hierarchical classifications well, but that I lost track of.) This would indicate that anything in either of the two Soccer subfolders was a member of the Soccer folder -- or was tagged 'soccer' if you like.

In fact, you can make a smart folder Soccer consisting of all the things which are either in Soccer vs Wayland or Soccer vs Arlington. You have to make it as a smart folder, which is not as intuitive, but woks fine. It doesn't give me the nice hierarchical user interface.

Labels

Actually I now want to associate some exportable re-usable data. The folder names are essentially my local tags. Exporting them doesn't help much.

Suppose, for example, I want to geotag the photos, so that I can find them on a map, or people interested in sports at the given field could find them. The current user interface allows me to select all the photos in one folder and apply keywords and apply metadata to them, as a batch operation. It is actually useful that the data is carefully stored in each photo, but it is sad that the fact that the metadata (such as a comment about the game) was applied to everything in the folder.

I'd like to be able to associate the random tag name I just made up with properties to be applied to each of the things tagged. Suppose at the user interface we introduce a label. A label is a set of common metadata that I want to apply to things at once.

The user interface could really milk the label metaphor, by representing a label as a box with a hole in the end with a bit of string. It clashes perhaps with the folder metaphor. If we use both, then I'd like to be able to drop a label on a folder, and let all the things in the folder inherit the labeled properties.

I'd like to see for each photo firstly what properties it has, but secondarily which labels and hence folder the properties came from.

The essential thing about a label is that as I build it, I am prompted to use shared ontologies. They could be group ontologies which others have exported, they could be globally understood ontologies like time and place, and email address of a person depicted. As I create the label from an (extendable) set of options in menus, and using drag and drop and other user interface tricks for noting relationships, I am creating data which will be much more useful than the tag. The tag then I can slap on very easily.

The hope is then that by making label creation something which is low cost, because I have to do it only once and can apply it many times, the incentive for me @@

Expressing labels

In this section we leave the user interaction and discuss the way in which labels can be exchanged in RDF under the covers. This of course is important for interoperability. A label can be expressed in many ways. in bits on the wire. The label describes a set of things, which in RDF is a class*. Information about the class and the things in it -- the things labeled -- can be given in various ways.

As a rule

As a rule, it could look like

   { ?x a soc:SoccerWaylandPhoto }
=> { ?x geo:approxLocation [ geo:lat 47. geo:long 78 ];
        foaf:depicts soc:ourTeam.
   }

In OWL

A label is a fairly direct use of OWL restrictions:

SoccerWaylandPhoto rdfs:subClassOf [
    [ a owl:Restriction; owl:owl:onProperty geo:approxLocation;
      owl:hasValue  [ geo:lat 47. geo:long 78 ]],
    [ a owl:Restriction; owl:onPredicate foaf:depicts;
     owl:allValuesFrom soc:ourTeam].

(Let's not discuss the modeling of depiction here, rather elsewhere.) This is very much the sort of thing OWL is designed for.

How not to

There is one trap which one must beware of. Remember that the label is a concept. It is a class. It isn't a photo. The label may have been created by someone, at a particular time, but that person and that time have nothing to do with the creator and time of a photo which is so labeled. You can not write

soc:SoccerWaylandPhoto
        geo:approxLocation [ geo:lat 47. geo:long 78 ];
        foaf:depicts soc:ourTeam.

Special vocabulary

It is possible to make a special label terms which are only used only for labels:

soc:SoccerWaylandPhoto
        LAB:approxLocation [ geo:lat 47. geo:long 78 ];
        LAB:depicts soc:ourTeam.

and have some metadata like

foaf:depicts  ex:labelPredicate LAB:depicts.
geo:approxLocation ex:labelPredicate LAB:approxLocation.

and a general rule like

    { ?x a ?lab. ?lab ?p ?z. ?p ex:labelPredicate ?q }
 => { ?x ?q ?z }.

or

    { ?lab ?p ?z. ?p ex:labelPredicate ?q }
 => { ?lab rdfs:subClassOf [ a owl:Restriction;
       owl:onProperty ?q; owl:hasValue ?z] }.

These methods are more or less inter-convertible. There are various communities which understand OWL and N3 rules, which may find those forms most convenient.

Sharing tags and labels

The architecture of this system then is that tags are initially local to the user. Anyone can use any word to to tag anything they want. Labels are used to associate meaning with them, but the tag itself is local.

Mapped into RDF, tags are classes in a local namespace. They can of course be shared. Tagging things with other people's tags attributes to them the properties associated with those tags, if any. Some people may define tags with rather loosely defined meaning, and no RDF labels, in which case others will be less inclined to use those tags.

Smart Labels and one-variable rules

When one combines a selection expression of a 'smart folder' with a label, then the result is a form of rule which is restricted to one variable. This can be expressed in OWL as a subclass relationship between restrictions.

A lot of information can be expressed as rules, but finding an intuitive user interface to allow lay users to express their needs with rules has been a stumbling block. These smart folder and label metaphors, combined, could be a route to solving this problem*.

Work in the area

There are many systems which use selection rules to define virtual sets of things. There probably lots which use an abstraction equivalent to labels.

One system which effectively uses labels is (I think) described as 'semantic folders' (@@link Lassila and Deepali), to be published

There is a language for labels being defined, as it happens, by the Web Content Labeling (WCL) Incubator Group at W3C. The final form of expression has not been decided.

Conclusion

The concept of a label as a preset set of data which is applied to things and classes of things provides an intuitive user interface for a operation which should be simple for untrained users.

There has been a lot of confusion from a wide varying uses use of this term for various different historical reasons, leading to uses which are sometimes ambiguous and in places inconsistent. This article attempts to shed light on the issue.

Historically, URIs were used to point to thinks like web pages and files and movies, on the web, useful documents, or "online resources" in the sense of useful things out there. FTP. Gopher and HTTP sites served up various types of online resources. People got used to http://example.com/ being a web page and http://example.com/#contact being an anchor within it.

The Online Information community, into whose domain the web stuff was put for standardization at the IETF, referred to these things like web pages as resources, and changed the original "D" for "Document" in "UDI" to "R". Some felt that resource was more appropriate term, maybe because "document" wasn't wide enough to include things like movies.

Now the URI spec actually allowed URIs for completely different things, such as telephone end points, and wisely the URI spec does not make any arbitrary constraint on what a resource should be, especially a resource denoted by a URI in a new scheme to be invented.

Meanwhile, the HTTP spec was polished and elaborated basically as a document delivery system, plus other methods for updating documents, plus POST. (POST started historically as a way of introducing a new web page y posting it to a list, just as in NNTP. It then almost immediately got used as a catch-all extension method. I will ignore it in this overview).

There was no real definition of what a resource or document was -- maybe because it seemed obvious. The HTTP spec did not even specify whether the URI denoted a person or a document about them, it just explained that the thing returned representation of the resource.

Roy's REST work then came along to formalize HTTP as REST and declared that a resource was a time-varying mapping between URI and representation. That was good enough for HTTP. It didn't have enough for the AWWW, when it came along, to be able to describe how the web worked.

In fact, the AWWW document, to explain how to use the web properly, had to add in a bunch of stuff about the social expectations -- things like, yes, the mapping from URI to representation is a function of time, but not just any old one -- a random function is not typically very useful. There are expectations about it can change with time. Persistence, consistency, with various common patterns which allow the web to be a useful medium. The AWWW decided to use the term "Information Resource" for a thing like a web page which contains information, and "Resource" for any old thing at all.

So HTTP and the REST work of was done very much in this space of document delivery, editing and update. There was no philosophical need to talk about what he URI denoted (the person, the web page about the person) until RDF came along, when there was an immediate need.

When RDF was first developed, it was motivated by the need for data about resources very much in the online information sense: data about documents, or 'metadata'. In fact it was designed to be able to describe anything, but many early users of RDF referred to it as metadata technology. RDF used the word "resource" rather awkwardly in fact as it turned out. In the beginning, many of the things being described were documents, and so the online information meaning of resource made sense. But in fact in RDF the resource was allowed to be anything at all. A class, rdf:Resource even used the term as the universal class of all things. A little later, the Web Ontology Language decided to use Thing for that.

RDF came along in what I think was a neat way. It used completely existing web protocol extension devices to introduce a new system which was fundamentally different from the old HTTP+HTML one. The HTML web was a hypertext model, which pages and anchors. The RDF model was a knowledge representation one of arbitrary things. It did this by using the fact that a new language can define whatever it likes as what a local identifier denotes. A graphic language might use local identifier to denote lines and points. HTML used local identifiers to identify hypertext anchors. RDF used them to identify arbitrary concepts, people, whatever.

The web architecture gave all these languages a common way of building a global identifier for the thing denoted by a local identifier in a given document. The semantics of the hash sign are defined web-wide to mean that "a#b" can be used to denote whatever is denoted by "b" in the document denoted by "a".

Worked a treat. At the beginning of the century, people played around and gave all kinds of things URIs like "http://example.com/ foo.rdf#color". Some of us did lots of work and made all kinds of systems which exchanged and integrated data in this way.

Two snags occurred, as the years passed. One was that a bunch of RDF users got the fact that it was good to use HTTP URIs, but didn't get the fact that you should put the foo.rdf online so that people can look up what #color means in it. And as they didn't do that, they didn't actually bother with the "#" at all. The second fly in the ointment was that some people wanting to use RDF for large systems found that they didn't want to use the "#". This was sometimes because the number of things defined in the same file was too low (like 1) or too large (like a million) and it was difficult to divide up the information into middle-sized chunks. Or they just didn't like the "#" because it looks weird. But for one reason or another people demanded the right to be able to use http://example.net/people/Pat to denote Pat rather than a web page about Pat.

This potentially led to huge failures in the whole RDF world, with systems already built which just used "http://example.net/people/ Pat" to identify the document whether you like it or not. I among others pushed back against using non-hash URIs for arbitrary things his but eventually gave in.

So in response to this, the HTTP protocol was, in fact, changed.

The spec wasn't changed. The spec editors were not brought on board to the new model. The spec was interpreted. The TAG negotiated in a way a truce between the existing HTTP spec, RDF systems, and people who wanted to use HTTP URIs without "#" to identify people. That truce was HTTPRange-14, which said that yoiu don't a priori know that a hashless HTTP URI denoted a document, but if the server responded with a 200 then you did, and you had a representation of the document. If you did a get on one of these new URIs which identified things were not documents (people, RDF properties, classes, etc) them the server must not return 200, it can return 303 pointing to a document which explains more.

So the HTTP protocol was, effectively, changed. The HTTP protocol as extended now allows HTTP to be used not only for Documents but for arbitrary Things. It extends the set of things which you can ask a web server about from documents to anything. It isn't a very bad design, nor very beautiful. Other designs would have worked, but that one was the only one which didn't have major problems for some community. It could be extended, but basically it works. It would be very expensive to reverse it in terms of systems which have been deployed.

It is also very expensive to go on debating it as though it is an open issue. It is reasonable to try to make the documents more consistent.

Anyway, that is a simplified version of the history of all this as I saw it.

I would like to see what the documents all look like if edited to use the words Document and Thing, and eliminate Resource. That's my best bet as to two english words which mean as close as we can get to what we want. Note however that the web is a new system, a design in which new concepts are created, so we can't expect english words to exist to capture exactly the concepts. So we take those nearby and abuse them as little as we can as far as we can tell at the time, and then write them in initial caps to recognize that that is what we have done

"A little semantics goes a long way", says Jim Hendler. He uses the phrase in lots of ways. I'm going to use it to say that the sense that a few ontological terms are so common that they will pull together all kinds of data ion all kinds of places. ___@@ at Mitre calls this "loose coupling: the idea that you would connect things through these few terms, and leave other connections to future projects. He talks about "Who, where, when?"

These are my ten terms for todays selected on the basis that they are important common dimensions, and if more data was connected using more of them then the data in the world would be a whole lot more connected. They are chosen from a pragmatic standpoint. There are probably

People

Let's start with "Who?" . I've picked two terms from the Friend of a Friend ontology because it is an ontology designed and used specifically in a bottom-up way to connect many people through nets of "knows" links between acquaintances.

foaf:mbox

The email address is the practical way most people on the net are identified. Web sites use email callback to establish that someone really does have the given email address. Yes, it isn't perfect, as

foaf:name

"When"? There are large numbers of quite complete ontologies of time, which use different models to discuss the complexities of the different scales, and of things like

cal:dtstart

cal:dtend

"Where" is a huge one.

geo:lat

geo:long

Documents are of course the lifeblood of the Web and the Semantic Web. The first call for structured data, which gave rise to RDF, was for data about documents. The Dublin Core group sat down to pick their top ten properties for docuemnts. Of the ones they produced, dc:title is the one I have found the most call for. I take the html <title> tag to have the same meaning.

dc:title

foaf:maker

In a fractal world, the total cost of ontologies for any project is reduced bny the fact that ontologies already exist in various commiunities

Tim BL 6 Feb 97

We can talk in the abstract about the sorts of things we'd like to do, but it is a wise thought experiment, to imagine how the user interface to a more powerful Web would be. So here is a an attempt to do that thought experiment. There are no screen mockups -- as I said, you have to use your imagination.  

Consistency

First of, all, the interface to a universal space should have a certain universal consistency. Currently, the user interface yu see on your PC or Mac has a few unfortunate inconsistencies which make it more difficult to use, and less powerful. One is that the "desktop" interface to the file system is different from the "browser". (See talk at Boston Web Conference WWW4, Dec 1995). It is crazy, if you think about it, that the whole screen is use to represent the information which happens to be on your local file system, using the metaphors of folders, while one window is used to represent the information in the rest of the world, using the metaphor of hypertext. What's the difference between hypertext and a desktop anyway? You can double click on things you find in either. Why can't I put folders into my hypertext documents? Why can't I write on the desk? Folders should be just another sort of document. My home page could be one, or it could be a hypertext document. The concepts of "folder" and "document" could be extended until they were the same, but I don't think that that would be necessarily a good idea. It's OK to have differet forms of object for distinctly different uses.

Location fixation

The only fundamental difference between a hypertext document and a folder is that there is a special relationship between any document and the folder where is "is". Unix allows a file to be in more than one directory (with hard links) provided they are on the same disk. DOS requires a file to be in just one directory, which again must be on the same disk. Users should not have to worry about what disks files are on. Suppose the system just files everything under its creator an d its creation date, and the rest of the system is just pointers (soft links, hypertext links, shortcuts), Then a folder becomes a useful sort of document for helping us organize things, but not a container with implications on physical storage. The relationship between a folder and a file in it becomes just a hypertext link. Ah, consistency!

Protocol - Smotocol

Another inconsistency is the current strange division between mail, browser, and news reader tools. Each have editors. The editors are in some cases plain text, and in some cases fancier things such as HTML. On the Internet, a mail agent allows you to use the Simple Mail Transfer Protocol, a news agent allows you to use the Network News Transfer Protocol, and a web editor allows you to use the Hypertext Transfer Protocol. This is of course totally meaningless to a user. From the user's point of view, the mail program allows you to move data between mailboxes (folders by any other name) and also allows you to link it into someone else's "In" box. I say "link" as it creates the relationship "this message (file by any other name) is in this mailbox (folder)", which we said above should be a link.

A news editor allows you to link a news document to a widely visible group (box, folder by any other name) which is visible to people all over the world. Functionally, it is very like mailing something to a list of people, as it creates links to the document from groups in each of their news readers.

A web editor allows you to upload a document into a web server, though the way in which you do that varies. (The original meaning of the HTTP "POST" operation was to have been a very equivalent "make new document and make link from this" operation.)

Now suppose you want to create a bit of information and you want to link it from a few individual's "in" boxes, a few news groups and a few hypertext documents. You also want it to show up in various folders. Which application should you use?

Clearly, this is a choice which the user should not have to make. Conceptually, a number of links are being made. In practice, various protocols will be used by the system. In the future, combined protocols may exist which efficiently perform all these functions as appropriate. Let's not bother the user with this. When I create an object, I want to pick the type of object I am creating. I also think it is reasonable, if anyone else is going to have access to the object, for me to specify the access list and the distribution terms: I am controlling my new intellectual property. It is also useful for me to specify what sort of quality of storage I want for the document. Do I want it archived reliably for posterity? Do I want it instantly available very rapidly and reliably? The answers to these questions will determine where the system will store my data. The last thing I want to be asked is the filename.

The "save as" filename dialog box is one of the things currently holding up our civilization. It doesn't ask the right information from the user. It asks it not when you are creating the document (and thinking about it at that high level) but when you have finished and are about to do (and thinking about) something else. [See Cooper on this]. As you move information from your head into a computer, everything can be intuitive until this step asks you to think of the disks and the operating system.

Of course, I won't want to be negotiating each of these parameters every time I create a new object, so probably I will have a set of templates, standard genres of document, for which these things have been set. In the case of HTML files, I will probably want to associate a default content and a default style sheet with each template. This will make it easier for me and for my readers to get an intuitive feel for the access and archival status of documents at a glance.

I will get back a URI from this operation. The quality of storage is part of the agreement between me, as a creator of an object, and the service I use to create and support that URI.

Sure, people like to pick URIs so that they can be mnemonic. I don't mind that. The problem is when the URIs are picked so that it is difficult to support them in the future. (See the section on naming, and HTTP PUT).

Let's assume that this inconsistency will be dealt with in future.

Signing: From Documents to Deeds

So the consistent user interface we have so far is one in which we are at home with documents and links, and we communicate by massaging the documents and the links.  The whole thing is "quasi-static" -- at any one time you can believe you understand a part of it.  It has state. When you change its state you can see what you are doing.

That's not enough.  It's not enough because in this model everything is malleable: every document is a "living" document. Everything can change.  This is great for an encyclopaedia but its no good for a check book.  It contains description, but because action is represented only in the limited constrained form to those actions which can be viewed as changes of state, there are things we just can't do. We can do idempotent actions -- those which are just as good if you do them twice as if you do them once.  This doesn't work for paying bills.

We can introduce actions which count (or if you like, "actions which you can count") in two ways which boil down to the same thing.  One way is for us to enrich the concept of operations on the net from "GET", "PUT" and "LINK" to a whole plethora of different functions.  We would probably divide were sources by their object "class", which would define what set of operations are available for each resource. This is the familiar world of distributed object oriented programming.

The second way is that we would allow, in the user interface, documents to be signed.  A signature on paper is a special thing (in principle).  It is a countable operation.  You make a signature on the document and it becomes something different.  No longer duplicatable at will, you act of signing is caught -- and you can be held to it.  The document becomes something which has been done: a deed. This is the familiar world of legal process.  On the web, it happens when you press a "submit" button and your order is submitted to the mail order company. When you make a document into a deed, it freezes.  You can't change deeds.  You can revoke them or cancel them out with other deeds, but you can't change one. Deeds are not living documents.  In fact, lots of documents are in practice frozen in that they aren't going to change much.  But they may not be deeds: noone has explicitly made an action on them.  

The two forms are equivalent, as when you make a deed, you can in the document write the name and parameters of any function you want executed, and submit it to a remote operation agent. Similarly, whenever a remote non-idempotent  operation is made using some Remote Procedure Call protocol, in practice the protocol involves making some message up containing the parameters, and directly or indirectly putting on it some sequence number or identifier to prevent it from being accidentally operated on twice.  Both are abstract representations of a commitment, and action which counts.

As we're talking about user interface here, I'd like to see a clean interface for making a deed, which makes it quite clear to me that I am committing something, and not just doing another search.  I suppose I have a set of "rubber stamp" icons which leave a name/date stamp on a document.  Different stamps can be made with different levels of security.  They may represent actions by different people, different roles, with different levels of authority.  I guess I'd have one for stamping W3C Recommendations which have been though the process, and a totally different one for ordering medium sized purchases on my credit card.

Deeds don't have to be signed digitally (but it helps).  Every time you press "send" in your email you are making a deed. The document freezes.  You may even be digitally signing it. You lose control -- you can't take it back.  

Socially, we will have to accept electreonic deeds. Also, we will have to define the limits of commitment which someone can imply by changing living documents without explicitly making a deed.

The "Oh yeah?" button

See also WWW4 Boston talk.

Deeds are ways we tell the computer, the system, other people,  the Web, to trust something. How does the Web tell us?

It can happen in lots of ways but again it needs a clear user interface.  It's no good for one's computer to be aware of the lack of security about a document if the user can ignore it. But then, most of the time as user I want to concentrate on the content not on the metadata: so I don't want the security to be too intrusive. The machine can check back the reasons why it might trust a document automatically or when asked. Here is just one way I could accept it.

At the toolbar (menu, whatever) associated with a document there is a button marked "Oh, yeah?".  You press it when you loses that feeling of trust.  It says to the Web, "so how do I know I can trust this information?". The software then goes directly or indirectly back to metainformation about the document, which suggests a number of reasons. These are like incomplete logiocal proofs. One might say,

"This offer for sale is signed with a key mentioned in a list of keys (linked) which asserts that tthe Internet Consumers Association endoses it as reputable for consumer trade in 1997 for transactions up to up to $5000. The list is signed with key (value) which you may trust as an authority for such statements."

Your computer fetches the list and verifies the signature because it has found in a personal statement that you trust the given key as being valid for such statements. That is, you have said, or whoever your trusted to set up your profile said,

"Key (value) is good for verification of any statement of the form `the Internet Consumers Association endoses page(p) as reputable for consumer trade in 1997 for transactions up to up to $5000. '"

and you have also said that

"I trust for purchases up to $3000 any page(p) for which `the Internet Consumers Association endoses page(p) as reputable for consumer trade in 1997 for transactions up to up to $5000."

The result of pressing on the "Oh, yeah?" button is either a list of assumptions on whcih the trust is based, or of course an error message indicating either that a signature has failed, or that the system couldn't find a path of trust from you to the page.

Notice that to do this, we do not need a system which can derive a proof or disproof of any arbitrary logical assertion. The client will be helped by the server, in that the server will have an incentive to send a suggested proof or set opf possible proof paths.  Therefore it won't be necessry for the client to search all over the web for the path.

The "Oh, yeah?" button is in fact the realively easy bit of human interface. Allowing the user to make statements above and understand them is much more difficult. About as difficult as programming a VCR clock: too difficult. So I imagine that the subset of the logic language which is offered to most users will be simple: certainly not Turing complete!

Specific notes on Windows UI and Typical Web browsers

(1997)

The gist of it is the need for greater consistency in the UI and the underlying system.

Consistency

Some basic principles:

1. Anything of any value and persistence must have a URI so that it can be referenced (yes, I know Microsoft have a Moniker scheme but now it has to be URIs to go global).

2. Any place I can use a URI I can use any URI.

3. Links are an evident as a primary user interface metaphor, with a consistent drag/drop or control/drag/drop for link creation, and consistent ways of viewing by link type.

4. The system should generate persistent URIs wherever possible. These can be just URLs in file or http space but they should not change. This is a longer term thing.

5. Things like start menus, bookmarks, favorites, recent document lists, toolbars, should be viewable as discreet objects in familiar ways. A Good Thing is the ability to easily see and manipulate the start menu in Explorer (right click Start). A Bad Thing is a modeful "organize favorites" dialog box is the most inconsistent outdated constraining way of moving things around I have seen in a while.  A modeless "Goto bookmarks" is better, but

I propose that any set or hierarchical structure should have a consistent windows explorer interface. You clearly feel that anything which is a set can also have an HTML view. In that case, I want HTML views of everything. Look at al the containers we have:

• Deskto
• Folder
• Start menu
• Toolbars
• Web pages
• Mailboxes
• Bookmarks
• Favorites

These must have consistent interfaces.

Some of these we can do without. Favorites, most of the Start Menu, and mailboxes are all ways of classifying things. They should all basically have the same interface and frankly I don't see the need for them being different. I want to put my inbox in my start menu. I want to be able to chose whether something I put in a menu is terminal or expanded. In general. (I could always put a link to the Favorites folder into the start menu but it wouldn't be expanded as a menu. I'd like general control over menus.

(Examples of things I want to do: Bookmark mail messages of interest, indeed any object.  Save a web page in a mailbox (which would freeze it in the cache and keep a pointer to it)

...

)

Dialog boxes are as we really know bad UI. The "Save As" and "Open" dialog boxes are a pain (I always want to be able to delete and rename files in them) because the are constraining and inconsistent, and they block until you get out. I'd be happy for an open box to simply launch an explorer window or select a current one, while providing a receptacle for icons which are to be opened. I'd like that receptacle (which is just the application icon) to be a URI I can save in any of the containers above.

The history list should of course be a something which works across all applications. A time-based view is a neat history lis, but it should apply across everything I have done in any applicationt. Make it generic so it works on any object. Make the index of everything I have done (maybe my whole click stream) be available as such an object, to be viewed just like a mailbox. I should even be able to make a link into it.

Maito: addresses

A mailto address is a misnomer (my fault I feel as I didn't think when we created it) as it is not supposed to be a verb "start a mail message to this person", it is supposed to be a reference to a web object, a mailbox. So clicking on a link to it should bring up a representation of the mailbox. This for example might include (subject to my preferences) an address book entry and a list of the messages sent to/from [Cc] the person recently to my knowledge. Then I could mail something to someone by linking (dragging) the mailbox icon to or from the document icon.

Modularity: MIME types and Operations

The OS as a concept  is currently having trouble with modularity -- the module used to be an application which provided its own communication and document types. Now the MIME types are orthogonal. I see new applications as either producing new mime type support or new link functionality (or both) but separably.

I should be able to mail anything. Suppose I am editing a photo. The tool bars are photo editing toolbars. WhenI want to mail it, I find the person I want to mail it to (any way: not from an adders book forcibly -- I can put my friends in my start menu or anywhere or of course find them on the web). I drag the document onto the person icon. Now that relationship gives me a choice (To, Cc, etc), and I will now get an extra toolbar for controlling the mail options. If I drag it to a newsgroup, the functionality will be exactly the same though the options may be different. I can then the links from the message by type (To, Cc, newsgroup, embedded hypertext link, version info, etc etc) in a consistent way. I regard dragging the icon of the document to a folder as being making a link too. The system should stop regarding folders as where things are stored. That has got to be decoupled, to separate the logical thing (I classify this as important travel) from the physical thing (I put this on drive C:). That is going to be a steady change, but early simple steps are

• allow the user to specify the algorithm for defining the filenames for new objects of each type, when defining templates, with a macro  like
  http://my.co.com/BY/<user>/<yymm>/NOTE/<seq>.txt
• give a good default set of macros which will result in filenames which don't have to change later
• ask when defining the template or creating a file without a template for a storage quality which will determine in the short term the filename  and in the long term a way in which the system will duplicate, backup and distribute the document
• ask at the same time for a default Acess Control List for the template.

When I hit "Send" or "Commit", then the document is frozen, and the  mail engines and NNTP software [and version control software]  starts to actual implement the links, and the web server allows appropriate access. All this is under the hood, which you can lift to see how its getting on if you like.

Archival and Access levels

This storage quality is a common parameter which I want to be able to change for anything. For example, I may decide that a web page or a newsgroup article I want to have on a "personal archive" availability. S I just change it with a menu item. No "Save As". No filenames. It has a URI already. And then I  know it will always be available to me on my desktop and laptop. Just like that.

The two toolbars of persistence level (think of name for that) and  access level are so important they deserve space maybe in that space to the R of the menu bar, or in window title banner. At least in icon form. I don't expect to have many combinations of them once I have customized the combinations I need with the archive and access wizard.

The archive status includes important flags for disconnected working -- is this object to be replicated and if so how often, etc.

Disconnected operation

The system has got toknow what its connectivity is at any time. Without a reboot! I would like this to be a switch available across the board, switching IPs and diconnected operation.

Interpreting HTTP

Browsers and Email programs are user agents. This isn't just a formal long term for them, it is an important issue. They are programs which act on behalf of, and represent, the user.

The computer protocols such as HTTP are defined to carry a particular meaning, and it behooves a user agent to representthat meaning to the user, or the whol system of peeople and machines breaks.

Here are a few ways in which browser designs should and often have not lived up to this.

Distinguish between HTTP "301 Moved permanently" and "302 Found"

There are two forms of redirection in HTTP. Each gives a new place to look for a resource, but for completely different reasons.

"301 Moved" is a response which indicates that the server has committed the unthinkable and for some reason not in a position to serve the document at that URI. It indicates that all references to the original should be change to the new one, including bookmarks and document links. This is an expensive solutin to a serious problem. It does not, of course, work completely, but it is the HTTP way for a server to alert a client of this situation.

"302 Found" is a response which indicates that the server is working as a name server. It is a success result indicating that you asked for a good document, and that the actual contents can curently be found at the given URI.

The imporant use of name server functionality is when for some reason it is impractical to server the document directly from a server which can hold the persistent URI. For example, a university might issue definitive URIs for its successful theses, and might have a very reliable, stable, low bandwidth machine which handles that URI space. However the content of the theses might in practice be delivered by fast machines located in the departments. The university may make a persistence commitment for the original URI but not for the department's server. Similarly, a user may for load reasons or speed reasons be directed to a mirror.

It is important that when a user agent follows a "Found" link that the user does not refer to the second (less persistenet) URI. Whether copying down the URI from a window at the top of a doucment, or making a link to the document, or bookmarking it, the reference should (except in very special cases) be to the original URI.

Very few browsers (Mozilla? Amaya>) implement this properly as of 1999.

There is also 307 temporary redirect, which is similar to a 302 Found.

See also:

• Cool URIs don't change (In the Style Guide for Online Hypertext)

Distinguish between an HTTP POST and a GET

GET operations (as happen when you follow a regular hypertext link) are fundamentally different from POST operations (as happen when you submit a form to order a book), The first is reversible, has no long term effect, cannot comit a user to anything. The latter does committhe user.

A graphic client, for example, should use a very different cursor while the user is hovering over a POST button to when the user is hovering over a GET link.

Doing a POST is like sending an email. (Currently 1999 it may be more secure because it will often happen over an https secure connection while many email clients do not encrypt messages.) It is really important to be able to find a list of the emails you have sent: these are the things you are committed to. The same applies to HTTP POST forms. The web client should keep a record of POSTs which have been submitted.

This would of course waste a lot of space for those web sites which get GET and POST muddled, but they are fundmentally broken anyway and the sooner we just fix this misuse on all sides the better. In the future, digital signature will be an action just like POST, but with weight added and the user awareness of the choice of key. Understanding when a commitment is made is a really important part of the user interface. Get it right.

See also:

• Axioms: HTTP GET if and only if no side effects
• HTTP specification

Hide URIs

Objective: A web user should never be aware of a URI while using the Web, either creatively or browsing.

Techniques:

• Hide all access to URIs inside special "under the hood" windows or status bars;
• Use titles to identify resources;
• Introduce RDF properties to indicate short titles and icons to make this easier;
• Remove all URI-aware functions to a special (optional) menu;
• Allow files for upload, or other documents to be referenced by drag-and-drop, or copy referemce/link to clipped

Web servers have to help by generating URIs for new documents. A new document creation form should redirect the user to a document whose UIRI has bene generatde, and which the user can then edit.

Solid’s core innovation lies in empowering individuals to control their data while enabling seamless connections between apps. From healthcare and education to finance, travel, and mental health, the integration of personal data promises tailored solutions, improved user experiences, and significant economic and social benefits. The Solid ecosystem fosters a new era of app development, democratizing access through “no-code” tools and encouraging the creation of self-governing communities.

By applying Metcalfe’s law to interconnected Solid Apps, the article highlights the extra value generated. It emphasizes the liberation from current internet constraints, such as siloed platforms and limited data portability, enabling more effective collaboration and problem-solving at both individual and community levels.

The article underscores the economic potential of Solid-driven innovations, not as an end goal but as a byproduct of enhanced productivity and creativity, collaboration and compassion. It envisions a future where empowered individuals and communities of all sizes work together to tackle global challenges, marking a paradigm shift in how we use technology to shape the world.

Read whole article...

Program Integration across Application and Organization boundaries

Introduction

Web Services mean many things to many people. In the end, there will be a set of standards which allow us to do things we could not do before, but in the mean time different people and companies approach them from different positions, and with different expectations. In 2001-2, Web Services have also been a buzzword used repeatedly and claimed to be one of the hot new technologies. The common themes are:-

• A departure from the web as a quasi-static information space to one in which interactions are the primary model
• A use of HTTP, XML and other standards from the web architecture as the building blocks
• A typical focus on enterprise wide and inter-enterprise operations

The Web in Web Services is, from the first point, a misuse: the term Internet Services would be more appropriate. The Web comes from the second point - the use of the HTTP and XML is already in use as a well-understood and well-debugged set of protocols which support the Web, and so it makes sense to reuse them in providing remote operations and those things connected with them. The third point is what makes web service requirements so different from a local RPC system. The fact that data is exchanged for business purposes and between different social entities means that accountability is required, rather than just reliable transmission.

• The vendors of software see web services as way to repackage existing capability in a way which makes it interoperable with other systems.
• The security requirements for web services are dictated by the trust environments, whether it is intranet or b2b or b2c, etc
• For b2b one needs not just reliabilioty but accountability.

The architecture of Web Services is the scope of the W3C Web Service Architecture Working Group.

Philosophy

Other articles have dealt with the fundamental architectural difference between remote operations and the architecture of the information space, and the mappings between the two.

• Axioms of web architecture (1990s) talks about the information space concept
• Paper Trail- Discusses the relationships between two patterns: read/write state derived from read-only documents in real life. Which came first, the journal or the database?
• Conversations and State (1998) discusses the trends in many areas away from shared information spaces, from Web Services to Voice browsing.

The Web Services architecture group has produced various drafts:

• Web Services Architecture Requirements (2002-11-14)
• Web Services Architecture (2002-11-14)
• Web Services Glossary (2002-11-14)
• Web Services Architecture Usage Scenarios (2002-07-30)

The architecture uses the following diagram for the highest level:

However, the essential part of Web services is the Interact relationship between a Service provider and Service requestor. This is the Web Service. Discovery agencies need not be used - they will in some cases but not in others. The discovery agencies are well represented as a cloud, rather than being a well-defined module in the web services architecture. They will become interface to a huge world of data and query services which provide data about web services as well as many other things.The Interact between between requestor and provider is the essential defining element to web services. As we shall see, the metadata about web services

Technologies within the Web Servcies umbrella

There is a mass of different pieces being bolted onto the foundations of Web Services provided by WSDL and SOAP 1.2 and th diagram implies things considerably. The management layer is a supervisory layer allowing the conrol of the many agents involved in a web services-based operation. The "Application semantics" layer indicates the necessity, for any useful interoperability, to have

Run Time messaging

The design work of web services is divided between the run time protocols and the descriptions of services.

The W3C work at runtime based on HTTP transport of XML-encoded messages, using the SOAP protocol. (Here by SOAP we mean SOAP 1.2, previous versions including early proprietory submissions which are not standards or guaranteed to interoperate) . There is a bifurcation in the design at this point, as SOAP operates basically in two modes.

In one, the XML message is used to encode the parameters to a remote operation in much the same way as remote method invokation in for example, Corba, DCOM, or RMI. In this mode, XML is used as the marshalling style, but the system is a distributed using remote procedure call in a fairly traditional way.

• There is a standard marshalling syntax

  Interfaces between software modules have well-defined functions, which in turn have well-defined and typed input and output parameters

  Stubs (dummy routines which similate the remote procedure by a local one which communcates with the remote one) can be generated directly from the WSDl definition

• The remoteness can be transparent, making the design of a distributed system similar to the design of a program.

In the other mode, SOAP carries an XML document, and the task of the receiver is seen more as a document processing operation. This is less rigid than the RPC style.

• The interface a service provides is defined just by the XML schema. This defines the acceptable document types, which can allwo extension in many ways, using XML namespaces.

  The communication is more apparent to the application writer, who deals with the document object model (DOM) of the recived message, rather than having parameters unmkarshalled automatically.

  XML tools such as XSLT and XML-Query, and XML encryption and so on can be used.

  It is simpler to use message exchange patterns other than the request/response.

The document mode of SOAP seems to be getting the most traction in the ecommerce stack. This is not an accident. The XML mode is more flexible than the RPC mode. It is easier in principle to extend an XML-based message system to include more information as a system grows. In fact, RDF is especially powerful in this area, as new information can be parsed into an entity-relationship form by old agents, and it becomes logically clear which parts can be ignored by those who do not understand them.

Functionality which has been mentioned as required above the basic layer at runtime includes:

• Routing. Routine data within message for processing bu different agents; defining workflow path of message. Black box or white box patterns of design.
• Security. Prolfiling existing security technologies for use in ebusiness applications using web sevices. Authentication and key management.
• Packaging of attachments to messages. XML Packaging.
• Reliable messaging (delivery, non-duplication, ordering) for the case in which the transport layer (such as TCP under the HTTP) doesnot provide this. (TCP does provide this reliability but (a) systems are not designed to keep TCP connections open for the weeks or years over which a web service may run, and (b) TCP does not provide accountability so you can show the tax man the acknowledgement of receipt 7 years later.)

Description

The descriptions of services are made at various different models and different levels of abstraction in different specs proposed as part of the stack, though there is agreement on WSDL as the modelling of the lowest level, the message or request/response interaction, and the binding to the specific HTTP (say) port at which it happens.

Higher layers in the description above WSDL are known variously as coordination, orchestration, choreography, composition.

They involve (compared with basic WSDL), for example:

• Protocols involving more than two messages
• Protocols having a common shared state over a long period
• Protocols having more than two parties involved; web service workflow
  • structured version (ws-*, damls process model)
  • precondition-postcondition style (DAMLS)
• The protocols as business protocols, in terms of common business functions
• The relationship between allowed transitions in the protocol and the content of messages. For example, the requirement for a transaction ID to match across a transaction, or for possible responses to be a function of a request code.

Composability and Choreography

Composability of web services refers to the building, from a set of web services, of something at a higher level, typically itself exposed as a larger web service.

Choreography refers more abstractly the part of the description of web services which defines a way, or the ways, in which a acyual invokations to various web services work together. (Peltz uses Choregraphy when it involves multiple parties, and Orchestration when it is internal to one party. Thus the former crosses application boundaries, the latter also crosses organization boundaries. here we use Choregorahy in general for both.)

There is so small amount overlap here, which has led to some confusion. To be general, one might say, for example, that a flight confirmation must involve an already reserved flight. This the actual constraint. One can describe a particular choregraphy (a particular dance, if you like) in which a flight query service is called, and produced a list of flights, and then a reservation service is called to reserve the flight, that is successful, and the resulting reservation is passed to the confirmation service. It may be that there are other ways -- other choreographies -- in which one could have achived a reserved flight. The engineer has the choice of modelling the many possible ways all in one choreogrpahy, or of making several choreographies.

Web services can be combined in such as way that messages are passed around in a very random fashion. However, a particular design techniqe is for a master process to delegate to other services in a recusive tree-like manner, as has been de rigueur in programming languages since Pascal. For example, if the consumer asks the travel agent and the the travel agent books a hotel, the hotel will reply to the travel agent, not to the consumer. This makes everything orderly.

WSCI, BPML and BEPL take this approach to choreography. This is a programming language approach with

• Sequential, Parallel and Exception execution, loops & conditionals
• Message-passing rendezvous between processes
• Calls: Web Services
• Data: bits of XML
• Assignment to variables
• Expressions: XPath 1.0 plugable in BPEL
• Does not handle actual calculations, rules etc.

WSCI has the empahsis on description, and BPEL on being able to compile to an executable agent. As neither is intended to do the actual calculations or business rules, it would be closer to compare themm with scripting shells such as bash which handle concurrency and synchronization but actually call programs (or rather web services) to do the real work.

See:

• WS Choreography Group
• IBM, Microsoft and BEA, under OASIS, BPL4WS (not W3C, not RF).
• BPMI, Business Process Modelling Language BPML
• Sun et al: Web Services Choreography Interface (WSCI), W3C Note
• IBM specs ws-coordination, ws-transaction, ws-orchestration

  Chis Peltz, Hewlett Packard, Web Services Orchestration - a review of emerging technologies, tools, and standards.

@@ Different attitudes - top down program design, or bottom-up agent design, bottom up document design.

Message-oriented choregraphy

The Paper Trail concept is that the state of a mult-agent multi-process system can be looked at, sometimes rather effectively, as a function of the documents which have been transmitted.

The process-oriented attitude to a bank-customer relationship may be "In parallel, the customer writes checks, merchants pay in checks, credit card transactions happen, all month. Then, the charges, interest are assessed and a bank statement sent from the bank to the customer". The document-, or message-oriented one is more like "Every month a bank balance lists valid transaction dated that month. A cleared incoming check in a valid transaction. A cleared outgoing check is a valid transaction. A validated credit card debit is a valid transaction. A check is cleared if it is incoming and there is a matching transfer from the payee bank", and so on. This builds the relationships up in a bottom-up, weblike way. The process-oriented attitude suggests the bank be written as a procedure in a top-down way using for example WSCI and BPL. The document-oriented attitude suggests the use of business rules systems triggered by the receipt of new information -- new documents, in this case new web services messages.

(Web service messages are of course documents just like documents sent in email. Messages are particular in that they have a particular time of transmission, and their document content sdo not change. They do of course generally have identifiers, and even though they can only be accessed by sender and explicit receivers, they can still be regarded as part of the web by those parties.)

Whether the design process is a top-down process-oriented one or a bottom-up document-oriented one, the design will have to be translated into a set of agents and their responses to incoming messages. This manipulation can of course be done automatically.

A concern in all this frantic design is it evolution with time. A BPEL script sets out to be a description of a a business process at a high level. The critical values which decide on conditional execution, or which correlate a particular process with a given transaction, are expressed as parts of the structure of the XML messages. This may lead to what has been called "DTD fragility". What happens which you change the DTD? The design of the message types with XML schema is the sort of thing which is difficult to get everyone in a company to agree on, and tents to change with time. There are many arbitrary choices made as to how the knowledge in the message is serialized as XML. Moving to RDF may, by removing a layer of arbitrary design, reduce that fragility and allow web sevice choregraphy to evolve with time within and outside a company.

Process modeling

When considering a business system with multiple agents and multiple concurrent processes, one would like to have an automated way of checking some fundamental questions.

• Will the process necessarily terminate?
• Will the service respond within a given time?
• Will the net gain from a sale always be positive?
• Will we ever promise to ship something we do not have in stock?

and so on.

The pi calculus and other calculi derived from it are formal ways of modeling systems with multiple agents and multiple processes. They can do some way to answering these questions. Rule-based systems can also be designed so that proofs can be found of these sort of conjecture. This is a good reason for keeping the languages involved as simple as possible It may be the design reason for the limitations on computational power in WSCI and BPEL.

See petri nets (IBM; and stadford), Pi calculus @@ refs.

Much of the functionality is seen in terms of tying web services down to well-known functionality such as exiting transaction processing systems, PKI trust infrastructure, and so on.

Discovery

In a large number of applications, web servcies will be provided by on one hand and used by on the other peers who have established relationships. Indeed, until a trust infrastructure is fairly developed it is not reasonable to expect computers to do automatic comparison shopping for very many services. Web services will probably (like the web in 1993, and the Semantic Web in 2003) spread first within the corporate firewall, where security problems are minor and mistakes less embarassing than inter-enterprise or publically. However, the goal is that so many web services should be available that it will be important to be able to find them in all kinds of ways.

The UDDI project and the related work on description and query systems is aimed at this. A positive aspect of UDDI is the definition of an ontology for web services. Problems with it are that it is centralized by design, both in the single-tree ontology, and in the design based fundamentally on a central registry, with inter-registry operation as a secondary thing.

From the semantic web point of view, web services are simply one aspect of the many things which will be searched for. Indeed, the fact that a web service is provided may in fact be rather incidental to the essential nature of the business item which is discovered -- a trader in stocks, a seller of lawnmowers, and so on. The semantic web aims to describe any aspect of anything, including the catalogs, parts, materials, services organization, relationships and contracts. A query system which addresses web services only makes sense when smoothly integrated with the rest of the web of enterprise knowledge.

Web Services and Semantic Web

The question of the relationship between these two activities is constantly in the air.

• The whole description side is a clear semantic web application, and so long as XML languages are defined which introduced with english language specs but no RDF mapping, there is a potential ambiguity which will have to be resolved later in making that mapping, there is an inability to use common semnatic web tools, and there is cost down the road assuming semantci web tools will eventually be used. Essentailly, web serv ices become instant legacy technoplogy for the semantic web.
• The DAML-services collalition of researchers is tackling the job of service description at a higher level.
• Many things which are described as web services can in fact be described as the publication of a series of semantic web documents, just as the billing of a peer company is in reality effected by the issuance of an invoice.
• When Semantic Web agents query each other, they could use SOAP (though a direct encoding into an HTTP URI may also be effective).
• When Semantic Web agents update each other, they should use SOAP, running typically over HTTP POST.

The argument against integration of the technologies is mainly social. It is costly to coordinate very large groups. It is much more effcient to develop WS and SW independently. Neither side has a great incentive to take on the learning required to absorb the needs and potentials fo the other. Using technology in preparation by another group takes a great leap of faith, and does really add to the development time. These are real issues. So while it may take more effort in the long run, it is a better parallelization of the design task to allow web services and semantic web to proceed in together without a mandated link. (This was the apparent consesnsus of the W3C AC meeting in Nice, 2000/11)

That said, wherever overlap of expertise between the technologies occurs, those who form a bridge should do their best to make the conceptual differences as small as possble. There is a Semantic Web Services group, connected to the ______

Service design tools

Most modern software design differentiates strongly between the design of an interface, and the design of the software which implements it.

Web services are required to be composable - you should be able to make a web service implmentation by building it out of component web services. At the low level, think of making a latittude/longitude to state code converter composed from a latittude/longitude to postal code converter and a postal code to state code converter. At a high level, think of a making a vacation being composed of resrvations of flights, hotels and entertainment.

Runtime System management

Real web services have multiple agents running commerical environments, in which downtime is expensive, and incorrect operation could be disasterous. The running, monitoring, provisioning and upgrading of such systems clearly requires tools, but their design is out of scope for this overview.

This discusses the introduction of URIs as names in a system to scale it to the web.

The web is extended in two ways - by adding new bits of technology to the existing stuff, and by "webizing" existing applications and systems. Webizing is really important, not only as a way of bootstrapping the web using large amount of legacy information, but because the existing systems have been researched and designed over the years and it is really important we do not lose the knowledge accrued during that process.

The essential process in webizing is to take a system which is designed as a closed world, and then ask what happens when it is considered as part of an open world. Practically, this effect on a computer language is to replace the names/tokens/identifiers for URIs. Thus, where before reference could only be made to something in the same document/program/module one can with equal ease make reference to something in a different one somewhere in that abstract space which is the Web.

In a clean case, this will be done so that the URI for an object is rather naturally related to its representation in the original language. For example, the element with ID "foo" in bar.xml is bar.xml#foo. However, to do the same for an attribute defined in a DTD or schema is more difficult, because of the complex nature of the spaces and subspaces for element and attribute names in XML. It is great when the webized language is very similar to the original language, and ideal when it actually compiles. Dan Connolly's 2000/8 webization of KIF uses URIs for identifiers, but to be accurate because URIs are case sensitive and KIF tokens not, lower case letter had to be marked with escaped with backslashes in the translation which made the result less readable. Changing the underlying language in small ways can make the translation much less cumbersome!.

Here is a slightly flippant view on the webize() function, each row of which probably needs an essay of explanation, but provided here without any.

Example - webizing a database

Imagine that a database is to be made available on the web in RDF. Suppose the database itself will have a URI of http://weather.org/current An SQL database is essentially a closed world, in that the various thing in it were not designed to be linked to from outside. An SQL statement

SELECT temp, zip  FROM weather WHERE temp  > 30

makes reference to terms which have meaning within the database. There is no reference in that statement to the database - that is simply part of the context.

Now suppose we determine what the URI will be for the pieces of the database, perhaps current/weather for a table, and current/weather.temp for a column in a table. We could then expend the syntax (excuse my SQL - I am making this up)

USING c FOR http://weather.org/current


USING u FOR http://places.org/usa


SELECT c:readings.temp, u:location.lat, u:location.long
  FROM JOIN c:readings, u:location
  WHERE c:readings.zip = u:location.zip
  AND c:readings.temp > 30;

This is an (incorrect I expect @@@) SQL which links out of the local database to combine it with information from a remote one. This syntax I am sure won't work in practice, but should illustrate the principle. Namespaces c and u are introduced for two reasons: for brevity, as repeating them in the code would have been too cumbersome; and for syntactic reasons as URIs tend to contain characters which would be ambiguous with other syntax is allowed in SQL column names.

Of course, whether actually SQL on a set of scattered databases is valuable may be questionable - it may not optimize as well as some other query languages. However, suddenly the things defined by the database are available to the outside world. For example, the concept of temperature reading as used by weather.org in its database of current conditions

http://weather.org/current/readings.temp

is now a concept, an RDF property in fact, which is available for all the world to refer to. These references need not all be in SQL. Because the schema for the database will declare it to be an RDF property or something equivalent, many different systems can use the information and refer to the concept.

Notes specifically on this example

I note, before we leave this example, that there are two concepts important to a table. One is the type of thing described by a row. A row in the reading table, for example, defined a weather reading, something which had a location and temperature and humidity and place. The other concept is the set of objects which are actually in the table. In the classic SQL example of the employees table, there is a rdf:class employee, subclass of person, and also the fact that someone works for the company iff they are in the table.

A second note on exporting databases. When you really put something on the web, there is often, for flexibility and security, a layer between what you expose and the internal storage. Just as web pages are not files though often closely related to files, and have the same form - a string of bytes and a MIME type. Exposed remote operations are not local procedures though closely related to them, and have the same form -- a service URI and a method name and parameters. Similarly one would probably export a derived view of a database in many cases - one which would have the form of a database. This allows different engineering decisions to be made on the external manifestation (persistent and what the customer wants) and the internal form (efficient and convenient for you).

Webizing nested languages

Sometimes this is easy and sometimes it is hard. It is hard, for example, when the language uses nested scoping to great effect. In this case there is a very large amount of context which is completely different between the beginning and end of such a link. The go to instruction is considered harmful [ref] by Dijkstra because it "as it stands is just too primitive; it is too much an invitation to make a mess of one's program." This of course is true of the hypertext link too, in a way. Both allow an open webbed world which typically, if used with no restraint, remove rules which give sanity and analysability to a language and allow optimization of the code compiled. So, just as some languages prevent one from jumping into or out of an inner loop of a program, so it may make no sense to allow a link to be made into something within a nested structure, because the referenced thing just does not have any meaning when taken out of context.

When dealing with language which have nested context, it may be necessary either to define how something inside represented independently of context, or to make it impossible.

Be careful, though, before jumping to this conclusion. In many cases, it is important to webize nested objects completely. For example, in a 3d scene language, an object may be within a scene within an object within a scene and still have identity which is important to be able to refer to. In a hypertext document, there is a nested context which for example affects the style, and the reference is made to the destination anchor not as a isolated piece of hypertext, but in the context of the whole document.

The principle that on the Web, anything must be able to say anything about anything means that these innermost nested objects must have URIs.

It may also be the case that an attempt to webize a language reveals bad points in the design which really need to be ironed out anyway for the cause of good software engineering. If a name in some module has in fact quite different meanings when used in different contexts, then it isn't suitable for webizing as it is, and maybe two separate derived URIs should be made in the mapping. Maybe the language should actually be cleaned up so that the concepts are distinct.

A very simple case is in a documentation control system, when humans use the same document name ("the pipe size draft") to refer to a particular document and also to the set of documents from

An exercise for the reader is to contemplate and determine whether it is webized, and if not, what it would take, and what would be the cleanest way of going it. Try looking at XML schemas (what is the URI of an element type?).

When stuck, recourse to common sense. Ask what the construct actually represents in a global context, if anything. This might mean clarifying the language itself.

Conclusion

Webizing a language involves turning from a system which assumes a closed world to one which will operate as part of the open web. Some cases are easier than others. Webizing one application gets one a good idea of what sorts of design decisions force a closed world assumption and make webizing difficult, and what by contrast makes a weblike application which immediately benefits from the rest of everything out there.

By accessing rich, structured, and personalized data, Charlie provided responses far superior to generic AI systems, as shown in a use case involving running shoe recommendations tailored to Bob’s fitness and lifestyle data. This personalization exemplifies the next level of AI capability, offering unprecedented usefulness while maintaining user trust.

We underscore the critical role of robust data infrastructure, including Solid Pods and the semantic web, in driving AI systems, and the dual role of AI in both populating and leveraging linked data stores, paving the way for a dynamic ecosystem where data graphs mediate AI interactions. We paint an exciting vision for integrating personal data, semantic web principles, and advanced AI to create tools that truly serve users.

Read whole article...

Abstract:

It might seem that the specifications of different XML namespaces can make inconsistent claims such that the semantics of a mixed namespace documents are inconsistent. The solution sometimes proposed is a "processing model language" such that there is no default meaning of an XML document without such an external processing definition. This article argues that there is only one basic generic processing model (or rather, algorithm for allocating semantics) for XML documents which preserves needed properties of a multi-namespace system. These properties include the need to be able to define the semantics of an XML element without clashes of different specifications. This introduces the concept of an on of an XML document is defined starting at the document root by the specifications of the element types involved. A common class of foreign element name, called here XML function, has to be recognized in default processing by any supporting application, and returns more XML when it is elaborated.

The problem

If one party sends another an XML document, how does one say what it means? Or, if you don't like the meaning word, what specs are invoked in what way when an XML document is published or transmitted? This question is sometimes posed as: What are is the processing model for XML?

The interpretation of a plain XHTML document is fairly well understood. The document itself is a human language document, and so the conventions - sloppy and wonderful - of human language define how it is understood and interpreted. And the interpretation of tags such as H1 is described in a well-thumbed standard and many books, and is implemented more or less consistently in many devices.

But what happens when we start to mix namespaces? When SVG is embedded within XHTML, or RDF or XSLT for that matter, what are the rules which ensure that the receiver will understand well the intent, the client software do the right thing -- and the person understand the right thing? The same issues obviously apply when the information has machine-readable semantics.

As Paul Prescod points out, there are plenty of places one might think of looking for information about how to process a document:

1. DOCTYPE statement
2. top-level namespace
3. schema reference declaration
4. other root-level declared namespaces
5. any attribute on the root element
6. anything in the document

In fact the general problem is that without any overall architecture, one can write specs which battle each other. "The X attribute changes the meaning of the Y attribute", "The Z attribute restores the meaning of the X attribute irrespective of any Y attribute" and so on. In such a world, one would never know whether one had correctly interpreted anything, as there might be somewhere something deemed to change the meaning of what we have. Clearly this way lies chaos. A coherent architecture defines which specs to look at to determine the interpretation of a document. We don't have this yet (2002) for XML.

However, in practice if a person were to look at a document with a mixture of XHTML and SVG, they would probably find its meaning unambiguous.

In the same message, Paul opines, Top-down self-descriptiveness is one of the major advantages of XML and I think that doing otherwise should be deprecated. I completely agree with this conclusion. He concludes correctly that the root namespace (the namespace of the document element) [or a DOCTYPE, which I will not discuss further] is the only thing one must be able to dispatch on.

The Pipeline Processing model

However, he secondarily concludes that, because it is important to define what processing to be done first, one should use wrapper elements, so that if there are any XSLT elements within a document, a wrapper causes XSLT processing to be done, and so on. The discussion about documents with more than one namespace has often made an implict assumption that the XML is to be processed in a pipeline, in which each stage implements one XML technology, such as include processing, style sheet processing, decryption, and so on.  The point of this article is that  while this works in simple cases, in the general case the pipeline model is basically broken.  Once you have things arbitraryily nested inside each other, there is no single pipeline which will do a general case.  And nesting things inside each other in arbitrary ways is core to the power of XML.

Specific cases: XML functions

The pipline model makes it very messy to address a situation which is increasingly common. This is of an XML document which contains a large numbers of embedded elements from namespaces such as

• XSLT, in "Literal Result Element as Stylesheet" mode.
• XInclude
• XMLEncryption
• XQuery (?)
• Internationalization tags such as "do not translate this phrase when translating this document"

These namespaces share common properties:

• They are the sort of thing you want to use with any sort of document, without it having to be foreseen in the schema for the original document
• The content of these elements is not the final form, but will be replaced with other content
• The resulting content may recursively have invocations of the same or different things from the list
• The effect of processing the element in this namespace is constrained such that it can only elaborate the contents of that branch of the tree. The element is replaced with its result of processing, but none of its ancestors or siblings may be affected.
• There are certain very special cases in which you want to be able to mention one without it being expanded.

To treat these as a group, I will call these elements XML functions. The term is not picked randomly. Let's look at some examples, each of which has its peculiarities.

XSLT Literal Result Element as Stylesheet (LRES)

Let me clarify this way of looking at XSLT. The XSLT spec defines an XSLT namespace and how you make an XSLT document (stylesheet) out of it. Normally, the style sheet has xsl:stylesheet as its document element. However, there is a special "Literal result element as Stylesheet" (LRES) form of XSLT, in which a template document in a target namespace (such as XHTML) has XSLT embedded in it only at specific places.  Here is an example from the spec.

<html xsl:version="1.0"
      xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
      xmlns="http://www.w3.org/TR/xhtml1/strict">
  <head>
    <title>Expense Report Summary</title>
  </head>
  <body>
    <p>Total Amount: <xsl:value-of select="expense-report/total"/></p>
  </body>
</html>

The XSLT spec formally defines the LRES form as an abbreviation for the full form. In doing so it loses the valuable fact that in the LRES form, XSLT elements behave as XML functions. They actually adhere to the constraints above. This is is very valuable. The XSL spec says that the interpretation be that an XSLT document be generated and processed to return the "real" document. However, this does not scale in design terms. As the XSLT specification itsels notes,

"In some situations, the only way that a system can recognize that an XML document needs to be processed by an XSLT processor as an XSLT stylesheet is by examining the XML document itself. Using the simplified syntax makes this harder.

NOTE: For example, another XML language (AXL) might also use an axl:version on the document element to indicate that an XML document was an AXL document that required processing by an AXL processor; if a document had both an axl:version attribute and an xsl:version attribute, it would be unclear whether the document should be processed by an XSLT processor or an AXL processor.
Therefore, the simplified syntax should not be used for XSLT stylesheets that may be used in such a situation"

It does not work when other namespaces use the same trick. It also prevents applications from using optimizations which result from the constraints above. So, while the spec formally defines a template document in that way, one can make, it seems, a completely equivalent definition in terms of XML functions.

Imagine a document in which at various different parts of the tree different forms occur, and in which these xml functions are in fact nested: you resolve an XInclude and it returns something with XSLT bits in.

It is essential primarily to define what such a document should actually be when (for example) presented to a user. It is an extra plus to have some visibility from outside the document as to what functionality will be necessary to have to fully process the document, such as from the MIME header, but we can get to that later.

XInclude

This is probably a simple function. The include element is replaced by the referenced document or part of document. This is straightforward and obviously nests.

It is also obvious that it doesn't actually matter , when xincludes are nested, that it doesn't make any difference whether you consider the inner ones to be expanded before or after the outer ones. (The base URI of a reference always has to be taken as that of the original source document, no matter where the refernce ends up being expanded)

Top-down Processing model

I think that the battle over the order of processing of XML functions is often an ill-formed question. XML is a tree. It is appropriate for the interpretation of the tree to be defined from the top down. This does not determine the order in which the leaves of the tree have to be done.

Here are some ways in which processors could handle an XHTML document containing XML functions:

• Noting that XHTML is a plain vanilla language, but that this document contains other things, first pipeline it through an XSLT processor, then an XInclude processor (the order being arbitrary), then a an XML decryption processor, and again in a cycle, until there are no functions left.
• Invoke an XML support class which then parses the document recursively. This more powerful XML parser has the ability to dispatch to the support class for an XML function whenever it finds one.
• Invoke an XHTML support class which then parses the document as it needs to in order to display it.. This more powerful XML parser has the ability to dispatch to the support class for an XML function whenever it finds one. However, the XHTML parser uses the constraint that in certain cases the front of an XHTML document can be displayed before the last has been parsed, and it actually delays evaluation of functions until the user's use of scroll keys makes it necessary. It turns out that certain things never need to be evaluated at all, saving time and bandwidth.

This is NOT supposed to be a definitive list of ways of parsing XML documents with functions - it is only supposed to illustrate the fact that many approaches are possible which can be shown to be mathematically equivalent in their effect. (This is why I tend to talk about the meaning, or interpretation, of a document, rather than the processing model)

The need to quote

That said, it may be necessary to define a reference processing model, just because one has to have a way of describing what the document means. In this case note that the first model above is not appropriate. It uses the fact that XHTML contains no tricks - it is "plain vanilla" in that everything in the document is part of the document in the same way, modulo styling. (I simplify). This does not apply to other sorts of document. Take an XML package for example: the contents of the packages are quoted and is not appropriate just to expand the contents of them. Only the cover note, the defining document contains the import of the package as a whole, and the interpretation of the other packaged things is only known in as much as the cover note defines it. it is essential that languages such as XML packaging can be defined in XML. It is essential that one can, if you like, quote a bit of XML literally, and make up a new tag which says something quite new about the contents. Therefore, while it works with XHTML, and as Tim Bray says (TAG 2002/02/14) there are many applications which do "generic XML processing" such as trawling documents for links and use of language, there will be certain namespaces such as HTML and SVG for which that makes sense and and other such as XML packaging and Xml encryption, in which it won't. (On the semantic web case, the same applies, and was the cause in 2002 of much discussion in RDF groups because RDF does not have quotes, and the informal use of rdf:parseType="log:quote")

If you need another example, think about the XSLT insert which generates and XInclude element: It may contain what seems to be and even is an XInclude element, but should not be expanded as contents of the XSLT element.

The reference processing model must be then, that parsing of an XML document conceptually involves elaboration of any functions, and that processors must be able to dispatch based on namespace at any point in the tree.

The result of such processing is the document which should correspond to the XML schema, if the. There is normally no call for schema validation of the document which still contains XML functions. Systems which claim to be conformant to the spec of a given XML function mean that they can, in any XML document, elaborate the function according to the specification. As Jacek Kopecky says (2002/02/21), [...] by saying on the sender: "We expect the XHTML processor to be able to handle XInclude and therefore this thing is an XHTML document all right". We can't of course expect old XML processors to handle XInclude, but we can expect anything which claims conformance with Xinclude to do so.

Software designs for top-down processing

In object-oriented software terms, one imagines handing an XML tree to an instance of an object class which supports the element type of the document element. This then returns something as defined by the spec. (An HTML document conceptually returns a hypertext page, an SVG document a diagram, an RDF document a conceptual graph (small c small g)). The object may itself call out to a parser to get the infoset for its contents, and it may or may not call out to the XML function evaluator but whether it does or not is defined by its own specification. But XML functions just return XML which replaces them. And any XML applications which claim conformance to the XML function's spec should be able to accept this.

Unresolved issue: references to siblings

This note does not address many of the issues around the XML processing model.

There is a possible ambiguity when a function refers to the current document. In other words, though it is not allowed to change things outside itself, it may read outside itself. This (if allowed) would clear raise the question of whether it references the document before or after its own or other function's elaboration.

A related question is whether an XPointer fragment identifier should refer to the document before or after elaboration of functions. My inclination is to say after, because then you know that an XPointer into an SVG object will resolve to a bit of SVG. But there may be arguments the other way.

XML Digital Signature (I am told) specifically requires that the signature is done on the raw source of the document before XInclude. Without going into the relative merits of signature before and after XInclude and other functions, it is clear that there are cases when either would be useful.

The ambiguity of these references, like the problems in XSLT of generating XSLT stylesheets with XSLT stylesheets, stem from the lack of quoting syntax in XML.

MIME content-type labeling

@@This section is not complete. It has been covered more thoroughly by TAG discussions already. @@ link

An XML document is in most cases self-describing. That is, you don't need to know anything more that it is XML to interpret it. In email and HTTP applications, it is useful for the RFC822-style message to define how the body should be interpreted using the content-type header. All that is necessary, then, is that the content-type should indicate XML (text/xml or application/xml or anything with +xml) and a top-down generic processing is valid. (The algorithm for determining the character encoding is not addressed here @@ link)

While this is sufficient, it is however useful to be able to provide more visibility as to what is contains [Roy Fielding, Dissertation, Ch4 @@link]. The document element gives, in many cases, fundamental information about the resulting type of the whole document, irrespective of functions elaborated or plugins plugged in. For example, whatever the content, an xhtml:html document is a hypertext page. This means that some systems will represent it in a window and allows certain functionality. The operating system, if it knows this, can use icons to tell the user, before they open an email or follow a link, what sort of a thing it contains or leads to. Similarly, an SVG document will return a diagram, and an RDF document body of knowledge -- a set of relational data. So more than any other namespace used in the document, the document element namespace is crucial.

This is why the best practice is to publish documents with standard and therefore well-known document element types as a special MIME type. This allows an XHTML page to be visible as such from the HTTP headers alone. This allows smarter processing by intermediates, decisions about proxy caching, translation, and so on. It allows the content negotiation of HTTP to operate, allowing a user for example to express a preference for audio rather than video. This also allows systems which want to to optimize the dispatching of a handler for the document from the MIME type alone. A "+xml" prefix as defined by RFC____@@ should be used whenever the document is also a self-describing top-down XML document for which the top-down processing model applies. (The fact that a document is a well-formed XML1.0 document alone does not constitute grounds for adding the "+xml")

Simon St Laurent has suggested [@@ his Internet-draft, possibly timed out] that all namespaces used in the document be listed as parameters to the MIME type. This makes sense on the surface. It may not be practical or worth the effort. It is a lot of bits, and in any case exactly what will be required cannot be determined until the document has been interpreted top-down. However, it or something equivalent is necessary if one is to specify the software support which is necessary.

• The top element can in fact be such that the other elements are to interpreted in arbitrarily weird ways
• For many document element types, there is a guarantee of the sort of object which is being represented.

Related notion: implied namespace

When a namespace-specific content-type has been specified, is it also necessary to specify the document namespace, or could that be assumed? That would mean that a plain XHTML file would not need an explict namespace. It is tempting to say that the default namespace should default to that associated with the content type, but in fact the logical thing is for the document namespace.

@@Decision tree diagram - add

User-defined processing of documents

This document defines the basic interpretation of an XML document. There have been many suggestions of ways in which a complex and different order of processing could be specified, many of these mentioned at the workshop, and including Sun's XML pipeline submission. My current view is that such languages should be regarded themselves the top-level document which then draws in the rest of the document by reference as it is elaborated.

Server-side processing of documents

In the HTTP protocol, or email for that matter, the important interface which is standardized is the one between the publisher (or sender) and receiver. We concern ourselves with what a receiver can do by way of interpretation of an XML document published or sent. Any processing which has happened on the server or sender side in order to process that document is not part of the protocol. While XML functions may indeed be elaborated to form a document for transmission from another one, that is something for control within the server and so is not a primary concern for standardization.

When a document is in a pure fucntional form, it actually is an opmization whether the functions are elaborated by the server or or the client.

The requirements on Schema languages

• <its:info translate="no">x</its:info> can occur anywhere x can occur (for systems which support ITS)
• <its:info translate="no">x</its:info> can occur anywhere x can occur, so long as x is human-presentable content.
• <xenc:encrypted> ...</> is a function: it can occur anywhere, so long as XEnc is supported. It's processing will return XML mixed content which will replace this element.
• <svg:drawing/> can occur anywhere <xhtml:img /> can occur.

Because the mode of operation in which the content is evaluated with function processing is very common, it would be useful in a schema for example to indicate this mode, or, more practically, to indicate the exceptions. There are very few elements which don't elaborate their contents at the moment in the markup world, and so they should be the exception. (Many computing languages of course reserve special punctuation for this quoting but adding punctuation at this stage isn't the XML style!)

Conclusion

The top-down processing model for XML as an architectual principle resolves many of the questions which remain unanswerable with pipelined processing. In fact, consideration of the example shows that pipeline processing could be actually dangerious, producing errors and possibly security issues, in the case of generally nested XML technologies of the types discussed.

References

• Discussion on www-tag@w3.org list
  • 19 Feb 2002, Paul Prescod Namespace Dispatching
• The archive of the XML-MIME list relevant to MIME dispatching of XML documents
• XML processing model workshop
• TAG Issue XMLFunctions-34
• W3C Specifications: XML spec; Namespaces in XML
• XML pipeline definition language, Sun Microsystems
• XML Processing Model discussion list (W3C members archive)