I’ve scribbled two pages of notes about Manx’s schema and philosophy which need typing up at some point. Well, “philosophy” is a grand word for it, but I mean some notes which explain why the database is organised a certain way, and how it is more than a list of documents. Manx is very much geared to cataloguing technical documentation, and a proper discussion of the features may help future maintainers.

The code that powers Manx needs some organisation and separating from the back-end scripts, many of which are obsolete and would only serve to obscure your understanding of the application. I’ll get round to that (hopefully) shortly.

The database is free for you to use in any way you see fit.

The most important reason is that I haven’t had time to maintain it for the last couple of years, and priorities in my life have changed so far that I don’t imagine I’ll ever get back to it. If I left the database online, it would only disappoint people as the catalogue covered a decreasing percentage of the available scanned documentation.

Unfortunately, when it was maintained, it was by a set of arcane scripts and command line wizardry that went straight to the database. There never was any decent graphical interface, and therefore there isn’t anything to pass to another maintainer. They’d either have to follow the same painful route as me, or spend a lot of time coding an interface.

I should have coded a graphical interface to allow people to contribute to the database a long time ago, but I don’t have much evidence of interest in its development. Many years ago, I received a few requests for copies of the database, and I sent them out. I also put database dumps online, so if anyone fancied building tools and feeding back the results, they could do so. However, I never heard from anyone who found them useful, or had any ideas for improvements.

Another reason for its demise is that Google is now OCRing scanned material, which will eventually make all of these archives searchable. Of course that isn’t the same as a catalogue, but the main purpose of putting documents online is to make them findable by, and useful to, more people. Cataloguing appeals to my desire to put things in order, but it isn’t a big thrill, as hobbies go. I’d rather work on attaining Shodan.

I’ve put the dump online, in case others find it useful: DEC documentation status.

This is likely out of date, because I don’t know what proportion of the DEC documentation online is not yet catalogued in Manx. I don’t even know what proportion of Bitsavers’ holdings are not covered, and I could at least calculate that.

I used to put fuller dumps of Manx online, and supplied complete dumps by email to a number of other archivists, but I no longer generate them as I never heard from anyone who’d found them useful. Nevertheless, if you’d like a particular report from Manx, please contact me, as it may help in planning the new facilities for users.

Manx lists the manuals produced by a bunch of old computer companies, and records scanned copies that have been put online. On the surface, the database is very simple. The records of each publication can be objectively correct; if you have the manual in front of you and the title, part number and publication date match the database, your work is done. However, Manx attempts to catalogue manuals that we don’t yet have copies of. These entries have come from documentation indexes, and are likely partial. Entries pulled from other databases or sources online are also likely to be partial, or contain errors.

At the moment, the database doesn’t store the history of individual records, as the assumption is that each record will become more correct over time, and there is no point recording how poor each entry used to be. However, collaboration changes that requirement. Even the best intentioned of contributors will make mistakes, and I now need a way of finding these and rolling them back.

I have been looking at different approaches at dealing with history in databases, and it is obvious that the plan of attack depends on why you need to store history. The term for this need seems to be “slow changing dimensions“. My application is rather like Amazon’s book catalogue. They accept corrections, and they go into the catalogue after a human reviewer has taken a look.

The approach I’m going to take is rather like the “Type 4″ methodology mentioned in the Wikipedia article, which happens to be the approach used for tracking changes to Wikipedia articles: history tables.

The current PUB table that keeps the details of each publication will be split into PUB and PUBHISTORY. PUBHISTORY will record every version of every publication record, along with some new details about when the change was made, and by whom. PUB will now store an index into PUBHISTORY for the current version of each publication record, to speed up searching. Once a row goes into PUBHISTORY, it will never be modified. Even deletion of a publication (because it was created by mistake) will be recorded at PUB level, rather than by deleting history.