Clearcase : VOBs database

Introduction to VOBs database

The VOB storage directory has numerous sub-directories in it. The four main areas of VOB storage are source pools, cleartext pools, derived object pools, and the database. Things can go wrong in any of these areas.

The VOB database (the db subdirectory) is one of the more important parts of the system (along with the source pools, which are in the s subdirectory). Taking a closer look at the database, we see that it uses the Raima proprietary format (where Raima is a company that's gone out of business twice since we started using
them). There's one database per VOB; on a VOB server, no matter how many VOBs you have, there will be just one database in each VOB storage directory. The database consists of 8 major files:

* Three data files (.d01 through .d03), which contain the actual data
* Four key files (.k01 through .k04), which contain indexes for random access into the data files
* String file (.str_file), which contains configuration records, among other things

Possible Problems and Their Causes

Three main types of things can go wrong:

01. Corruption in source containers : Source containers get corrupted for a variety of reasons, usually network-related. These reasons include: NFS problems that create zeroes in the middle of the files; network problems such as fragmentation or reassembly errors for large NFS packets; or faulty NICs (network interface cards).
02. VOB accesses not working : VOB accesses can stop working because you ran out of disk space or hit an OS file size limitation (2 GB on some systems) or internal database limitation (particularly in ClearCase 3.0 or schema version 53). If you encounter one of these problems, you're in big trouble, because you won't be allowed to write anything else into the VOB.
03. Data loss or corruption in the VOB database : Hardware-induced failures, Software-induced failures, from either the operating system or ClearCase (in Raima), try to remove individual database files to save space.

Detecting Data Loss or Corruption

Symptoms of having a corrupt VOB database include the following:

* You're unable to access the VOB or VOB objects. (cleartool commands fail, reporting an error like a database ID not found.)
* Either scrubber or vob_scrubber fails.
* You can't lock the VOB (which in general means you can't do any write transactions to the database).
* You can't replicate (or import replica packets). This indicates possible database corruption, but it might also just indicate divergence, depending on the error logs.
* Reformatting the VOB fails. In this case, the database is almost certainly corrupt.

The only two processes that talk to the VOB database are db_server and vobrpc_server. In their ClearCase log files, you'll see messages like these in the event of data loss or corruption:

* Unable to open VOB database - This isn't critical. It usually means you've tried to move or copy the VOB without maintaining the proper permissions on a subset of files in the VOB storage directory tree).
* db_VISTA error nnn- You'll see error numbers like -922, -909, and -6. Note that db_VISTA is essentially synonymous with Raima; anything that's a db_VISTA error is a Raima problem, although not necessarily a corruption. Raima could be having problems with the system, as in not being able to see it, read it, or write to it. This message can also mean that the API being used in ClearCase to interact with Raima is returning an error code.
* Internal error in VOB database, Unexpected error in VOB database, or Something not found in VOB database (basically, "I don't know what you're looking for, but it's not there") - If you see one of these not-so-useful messages, please call Rational Technical Support right away; don't wait a few months or ignore it altogether.

Detecting When a VOB Is Near a Limit

A VOB can stop working if it's near a limit. To detect how full a VOB database is, two utilities are available from Technical Support: countdb analyzes the data (.d01-.d03) files, and string_report analyzes the string file.