SetLink: the CERN Document Server Link Manager

This paper will stress the importance of using a Link Manager for any long term Web server. It will explain how the CERN SetLink Link Manager is designed to handle a wide range of document types and formats, from photos in JPEG to eprints in PDF. It will also focus on other possibilities offered by using such an application, like automatic figures extraction or concatenation, full text searching and on-the-fly format conversions.
 

Background: the CERN library and Document Server

A large repository of meta-data ... and of data

This vast repository of meta-data contains more than 255 000 preprints, 43 000 books, 1 200 periodicals, 9 000 administrative documents, 25 000 official archived records, 2 050 photos, 1500 press cuttings, etc., and it is always growing. The description of a document (its fields) is achieved using a customization of the very complete MARC [2] format, stored in the Aleph automated library system [3].

The CERN system  is also composed of a large quantity of data: electronic versions of full text documents (PS and PDF), paper versions which are scanned to be delivered electronically, eprints, scientific committee papers, academic training, photos, press cuttings, etc. The CERN Document Server stores today about 70 000 files, which represents about half of the total fulltext linking done from the Library system.
 

Two types of URL

As any Internet user will confirm, the trouble with URL is its very short lifetime. The so-called "U-N-F" - URL Not Found - problem can result from changes at three levels: the protocol (ftp -> http -> https) which is rare, the server name (Error 602: Connection failed) which is more often and the file names (the famous Error 404) which is so frequent.

In order to cope with this problem within the library system, we distinguish two types of URL, the stored ones and the derived ones.

• Stored URL: specific fields do exist in the database where complete addresses are kept.

They are considered as controlled URL if they point to the CERN document server or to some official CERN site (which will be maintained for the lifetime of CERN).
They are uncontrolled URL if they link to documents or pages outside the laboratory, or possibly to "wild" CERN servers. Of course, this last type of URL are the most problematic. In order to isolate broken links, the only solution is to run an URL Checker like MOMspider [5] which is run regularly until it can certify the address is broken. Correction or deletion must then be done "manually", one by one. The first time it was run on the book collection (summer 1998),  already half of the URL were obsolete.
In general, the stored URL are easy to collect and to check but their maintenance is a real pain, even with the help of software.
 
• Derived URL: these are addresses which are created on-the-fly from another information in the meta-data set.

It can be derived from a report number, from a publication reference, from a conference code or from any other key value. The link is generated by the program displaying the results of a search.
The main trouble encountered with this type of URL is that broken addresses are generally discovered by chance. No URL checkers can be run on "on-the-fly" URL. If an electronic journal changes the organization of its articles URL, or if it disappear,  all the links to it become dead links. The advantage of this technology is that a single maintenance action (e.g.: remove the e-journal name in the list of CERN e-jounals on-line) will be understood by the program, and all the linking remains valid. Sometimes, it may be more complex (e.g.: introducing issue numbers in URL to articles) and it then requires some maintenance at the programming level. Still, only one maintenance effort may correct thousands of links.
So, even if it is a bit problematic to check the persistency of derived URL, they are much easier to maintain than uncontrolled stored URL.

General solutions to broken links

Meanwhile, resolution services should exist on the Web servers themselves. Links to files should be systematically replaced by links to a resolution application (a link manager) in charge of fetching the requested page. Examples of such a technology are PURL (Persistent URL system) [7] or DOI (Digital Object Identifier) [8].

As the CERN library system is not only a "URL consumer" but also a "URL provider" - many sites are pointing to library resources or to full text documents - The CERN Document Server has developed its own link manager (called SetLink) which serves today the 70 000 full text documents and photos.  In this way, all links from the library databases to the CERN Document Server will be long term: independent from server names, files locations or files formats.

We will see in the next chapter what is the SetLink link manager and why it was decided to use an in-house link manager instead of a commercial alternative.
 

The SetLink Link Manager

SetLink is not a search engine, but for a given base, category and ID combination it should be able to return the document the user is searching for, and offer some additional services in return. You can, for example, convert the document files to other file formats such as PDF, PostScript or even into a GIF image page by page, perform keyword searches on PDF documents, or send the document to a local printer or to your email account, all from the same web page returned by SetLink.

Overview of the structure of WebLib, SetLink and CDS:

A Commercial Alternative

Other factors also needed to be considered, such as cost and functionality. It would be cheaper for Library Support to have developed SetLink for themselves, rather than to pay for a license to use a commercial Link Manager. SetLink would also function exactly as is required, and be much easier to maintain and customize if it were developed in-house.

SetLink also performs many "under the bonnet" functions which the end user is not aware of, such as making a record of links which are broken, documents which can be converted to other formats, as well as sending email to the Library Support team when other things go wrong. No existing Link Manager was available (in 1998) which would have performed some of the very specific functions that SetLink does.
 

A Little History Lesson

In 1996 WebLib was introduced on the Preprint server along with the first version of SetLink. This was a simple version of the program which offered the file for downloading, and a few simple file format conversions.

In 1998 SetLink was upgraded to version 2, which offered a lot more features as well as file format conversions. The differences between versions 1 and 2 were considerable as version 2 represented a complete system redesign and rewrite of the source code.

SetLink was upgraded further in 1999, though the changes made at this time were mostly cosmetic. All the HTML written by the program was tidied up and a cascading style sheet was implemented. The program was also optimized and made to run at a faster speed. Today, 15 different types of documents (bases) are handled by SetLink and 102 possible categories are declared.
 

Technical Description of SetLink

Secondly, the way in which SetLink writes all its HTML back to the browser was altered. Instead of having lines of code in the program to write generic HTML to the browser, the HTML content was removed and divided into smaller  files which were fragments of a larger page. The program would select a fragment  file for each part of the web page and parse it instead.
 

SetLink configuration file

The major difference in how SetLink behaved each time was dictated by the "base" parameter passed to it at run-time. This parameter dictates the content of the HTML written by the program, what methods the program uses to look for the files on the document server, which services were offered etc.

The configuration file was designed from the analysis of the differences between each base. It is laid out in terms of each base, with a series of parameters to define that base's behavior. When the SetLink program runs each time it checks the base parameter and draws its behavior from the parameters specified in the configuration file.

In addition to the base definitions the configuration file also holds useful information about the local file system, such as the software commands for performing the file format conversions, and the locations of other useful files.

Example base definition from the SetLink v2.1 configuration file:

   // Special case for EXT
   //
   base(preprint) {
      directory: /archive/electronic/cern/preprints
       category: ext
    shPrefixers: |/$p|/$m5i4|/../scan/$m5i4
        formats: pdf|ps.gz|ps
       bodyHTML: validHtml/pdf/body.html
      linksHTML: validHtml/preprint/links.html
       tailHTML: validHtml/cds/tail.html
      wildcards: _*|fig
    noCategHTML:
       noIDHTML: validHtml/cds/notAvailable.html
     noIDScript: /users/setlink/preprint.notFound.sh $1 $2 $3 $D
   }

SetLink's HTML

In order that the HTML be interactive with the program a few rudimentary codes were inserted into the HTML, now known as dollar commands because they are preceded with a "$" character. At first these were reasonably simple things such as $Fn for a filename. Hyperlinks to a document file are created thus;

You can download this document in the following formats:

Valid HTML and Style sheets

Limitations and Perspectives

File Organization

External factors

Perspectives

But whatever is the evolution of the technologies, the current use of the SetLink link manager is a guarantee of a minimal impact on the maintenance of the CERN library system and at the same time the assurance of the ability to quickly link all bibliographic notices to the best corresponding resources.

Finally, we are optimistic that more and more URL providers (like e-journals already involved in Cross References service [12]) will be using link managers in order to decrease the "Error 404" rate and to improve the quality of library services world-wide.
 
 

URL References