KWTR: Semantic Web benchmarking
[edit] Contributors:
Raúl García-Castro (Universidad Politécnica de Madrid)
- What is the state of the art of Semantic Web in your research field?
The Semantic Web technology has improved considerably since the development of the first tools in the nineties and, although it has been mainly used in research laboratories, in recent years companies have started to be interested in the Semantic Web technology and in the applications using this technology.
In order to consolidate this technology, both in the industrial and in the academic world, it is necessary that the Semantic Web technology reaches a maturity level where it can comply with the quality requirements required by the industry. Therefore, the Semantic Web technology needs both to be thoroughly evaluated for providing objective results and to attain a massive improvement in their quality.
Up to now, the Semantic Web technology has seldom been evaluated; however, as its use has spread, in the last few years numerous studies concerning its evaluation have appeared. Therefore, it seems necessary that researchers increase the quality of their evaluations and aim for collective improvements in their technology by means of benchmarking it using a methodological process.
The role of the research community when defining and performing benchmarking activities is crucial. Community-driven benchmarking connects experts and allows obtaining high quality results and increasing the credibility of the benchmarking and its results.
- Provide references and short abstracts of three papers you consider as significant in your research field.
- García-Castro, R. "Keynote: Towards the improvement of the Semantic Web technology". In Proceedings of the Second International Workshop on Scalable Semantic Web Knowledge Base Systems (SSWS2006). November 5. Athens, Georgia, USA (2006). The paper presents software benchmarking as a process to carry out over the Semantic Web technology in order to improve it and to search for best practices. It also describes a software benchmarking methodology and provides recommendations for performing evaluations in benchmarking activities.
- García-Castro, R.; Gómez-Pérez, A. "Guidelines for benchmarking the performance of ontology management APIs". 4th International Semantic Web Conference (ISWC2005). 6th-10th November, 2005. Galway, Ireland. LNCS 3729, 277-292. The paper presents briefly the benchmarking methodology used to improve the performance and the scalability of ontology development tools. It focuses on the definition of the infrastructure for evaluating the performance of these tools’ ontology management APIs in terms of its execution efficiency. It also presents the results of applying the methodology for evaluating the API of the WebODE ontology engineering workbench.
- Are there existing tools or demos? Please indicate some of them.
- What are the open problems in your Semantic Web research field? Why?
Although the number of evaluation and benchmarking activities is continuously increasing in the Semantic Web area, it is still not good enough to ensure a high quality technology. The activities carried out involve just a few types of Semantic Web tools and, in general, evaluations are not applicable to other types of tools. Furthermore, the reusability of and the accessibility to the evaluation approaches is not high, being this a hindrance to their general use.
- Provide references and links of the most relevant Semantic Web research projects in your field.
- What challenges try these projects to overcome?
The Knowledge Web project aims to offer a framework for comparing technological results and for measuring the progress of research on an objective basis.
- What are their foreseen benefits (both in market and scientific community)?
The main benefit is to improve the technology and to support the transference of this technology from research to industry.
- When, in your opinion, will projects’ results be ready for industry?
Project's results will be ready for industry when they are able of showing their value in a credible and objective way. This includes that industrial parties are able of evaluating these results on their own.
- Do you think that it is important to invest (money and time) in these topics? Why?
Yes, in order to achieve a successful technology transfer from the Semantic Web community to industry, it is required to perform tool evaluations or benchmarking activities easily and at large scale. These evaluation and benchmarking activities are time and resource consuming in their different phases: when organising the activity, when performing the experiments, and when analysing the results.
- What are, in your opinion, the most relevant Semantic Web challenges that will be solved in the long term (10 years)? Why?
The Semantic Web needs the production of methods and tools for evaluating the technology at great scale in an easy and economical way. This requires defining technology evaluations focused on their reusability.
However, future research must also focus on performing user-centered evaluations of the Semantic Web technology and on considering audiences from outside the research community as a recipient of the evaluation results.
