SysAdmin Adventures

If you are using incremental backups with Veritas NetBackup (whether differential or cumulative) you should be aware that by default NetBackup decides whether to back up a file based on its mtime or Modification Time. This means the time shown on the file if you do a normal ls -l. This may not be desirable if you are moving files between backup clients.

If, for example, you “rsync -av” a directory from one server to another, the modification time of the files will not change so any files last modified in 2002 will still have a 2002 mtime and therefore have not changed since the last full backup. This means NetBackup will ignore those files until your next full backup and potentially leave you without the backups you thought you had.

The solution is to set NetBackup to consider the ctime in determining inclusion in incremental backups. The ctime is the inode modification time, effectively meaning the time that information about the file last changed rather than the time the file itself last changed. You can check a file’s ctime with

A newly moved, but unmodified, file’s ctime will be the time it was moved to the filesystem (or the time file permissions or other metadata last changed after it was moved).

To do this, set the following in the client’s bp.conf (usually /usr/openv/netbackup/bp.conf):

For more information on this, see:
http://seer.entsupport.symantec.com/docs/200644.htm

This is a the text of a support case I have just raised with Symantec regarding administration of NetBackup Server 7.0:

Support has been dropped for 32-bit Linux clients. I appreciate that NetBackup Server will still work with v6.5 on 32-bit Linux hosts.

However, the java client for administering the NetBackup Server installation is not a 64-bit application, it works regardless of the environment so long as a valid JRE is available. However, the way it is distributed is such that it cannot be installed on a 32-bit Linux machine and I do not see why.

I do my daily work from a 32-bit Linux machine. This machine is not a backup client nor does it need to be. All I need is to be able to administer the NetBackup server. Up to now I have been having to remotely access the NetBackup server and send the display back to the X server on my 32-bit machine in order to administer the server. This isn’t really an acceptable solution.

Obviously, the first thing I did was to try the java administration client that comes with NetBackup v6.5, however much to my amusement, the 6.5 client refuses to speak to the 7.0 server.

This leaves me only with the option to use the 7.0 Java client on a 32-bit Linux machine.

I cannot use the cp_to_client script as it detects the lack of an x86_64 environment and dies. I cannot use the extract_java script because it lacks a client environment to install into. I could screw around with these scripts to bypass the checks but it’s a waste of time considering how little the java client actually is and would leave me with 64-bit client software I neither want nor need and a 64-bit JRE.

Bottom line:

I installed the 6.5 client software, removed /usr/openv/java and manually extracted NB-Java.tar.gz over the top. To provide a JRE I successfully tested both the JRE that comes with 6.5 (1.4.2_13) and my own installed JRE (1.6.0_18) and both work flawlessly.

HOWEVER.. I first had to modify jnbSA (and jbpSA not that I use it) to remove the setting of the PLAT64FLAG=”-d64″ variable.

Having done so, the client works absolutley perfectly on my 32-Bit Linux machine just as I would expect it to, being a java application.

So, the question is.. why have you done this? I can understand, if not condone, your wish to kill multiple architecture support for the Server and Client software, but regardless of the platform the backups run on, the System Administrator needs to be able to administer the system from wherever he/she works.

Since you already have a working java admin client that works perfectly on pretty much anything with a JVM installed, why are you going to such lengths to prevent it from running on a 32-bit Linux machine?

In order to provide this ability all you need to do is ship a 32-bit JRE as well as the 64-bit JRE (or even just allow a user to specify the location of his own JRE if he has a 32-bit client) and provide an installer that will install it regardless of the platform or whether the backup client software is also installed.

I could personally knock such a thing up in about 10 minutes, would you be prepared to?

The response from Symantec support:

As you know, the Java GUI is provided as a bundle for Linux and is not separated out into its own install. Because of the limitation of x86_64 Linux servers only that they decided to place on the product, the install script includes a check for x64.

At this time, Symantec does not plan to release an x86 installer version of the Java GUI for Linux. As you know, this isn’t due to any limitation, but it’s due to a decision made by the developers. At this time, I do not believe any plans are in place to create an x86 installer package for the Java GUI on Linux, however we can help speed up the process. We have a community enhancement request page http://seer.entsupport.symantec.com/docs/318008.htm where enhancements can be voted on and pushed into the product faster. This site is monitored by the Product Managers. The more votes an enhancement gets, the faster it gets implemented.

Since the only way this little mess can get sorted out is to have customer support for it demonstrated through the Ideas portal, I have again created an Idea on the portal and would appreciate any interested parties voting up the idea to bring it to the attention of the product managers.

https://www-secure.symantec.com/connect/idea/provide-separate-java-gui-installer-linux-include-x86-support

Let me start by clarifying that I’m actually talking about slackware64-current as of Tue Mar 2 19:07:31 UTC 2010, but for longevity I have titled the post 13.1 because I believe -current to be pretty close to a 13.1 release and I will be updating this server to 13.1 at release and halting updates there.

That said, Symantec have released (Veritas) NetBackup 7. Woo. As insane as it sounds, NetBackup 6.5′s Linux server component was only available for x86 and IA64, but NetBackup 7′s Linux server component is only available for x86_64. Way to screw everybody over, Symantec(!)

Thankfully, this just happens to be beneficial for me as I am at the point of being able to move our NetBackup server from Solaris10-SPARC to Slackware Linux and it means that I get to deploy it on Slackware64. In this case, the server I am using has had alienBOB’s Multilib Conversion; but this is not necessary to support NetBackup 7.

Perversely, the new “OpsCenter” (x86_64) that Symantec won’t shut up about does require 32-bit libs, however it’s also much more tightly bound to RHEL/SLES. NetBackup Server works well out of the box in Slackware, but OpsCenter won’t install on anything but RHEL or SLES and manually extracting the RPMs and forcing it in is much more work that I can be bothered with for a component I probably won’t even need. As far as I can tell, the main reason they are giving away OpsCenter is because they want managers to see it so that they can persuade them to buy OpsCenter Analytics which I’m certain will not be cheap.

That’s the ranting out of the way. Onto the details…

Installation really is a breeze. I used the RHEL version of the installer (NetBackup_7.0_LinuxR_x86_64_GA.tar.gz) not the SLES version (NetBackup_7.0_LinuxS_x86_64_GA.tar.gz) because I am more familiar with RHEL than SLES and I have always been successful in the past using Veritas’ RHEL releases on Slackware. Feel free to play with the SLES version and report back.

Extract the tarball and run the installer. I used the defaults for pretty much every option. Feel free to change the install path from /usr/openv to /opt/openv for correctness, but I am personally very familiar with /usr/openv (as is the software, obviously) and the software lives almost entirely in that subdirectory without polluting the rest of the machine; so it’s a safe choice.

The installer will modify the inetd configuration and add entires to your /etc/services file and put two startup scripts (netbackup and vxpbx_exchanged) in /etc/rc.d/init.d. It will also put symlinks to these in the run-level numbered /etc/rc.d/rc*.d directories. Worth remembering that under Slackware /etc/init.d is a symlink to /etc/rc.d/init.d. In my previous post about running NetBackup client software on Slackware (http://blog.tpa.me.uk/2009/08/30/veritas-netbackup-slackware/) I suggested that you should remove the symlinks and move the netbackup startup script to /etc/rc.d/rc.netbackup. Technically I still believe that advice as I am somewhat of a purist, but I can also be pragmatic when necessary. Slackware has a startup script deisgned to deal with SysV-style scripts called /etc/rc.d/rc.sysvinit and there’s no real reason you shouldn’t just let it do its job and leave the NetBackup scripts where they are. It will certainly help out when it comes to upgrade time; you just have to remember to use /etc/init.d/netbackup (shortest option) rather than /etc/rc.d/rc.netbackup.

Now it’s time for…
The bit that will catch you out:

As is widely known, Slackware still does not contain PAM. There are reasons this is good and there are reasons this is bad. Pat is well aware of all of these reasons and continues to consider his options. In the meantime, we have to be aware of the lack of PAM and work around it where necessary (assuming you don’t choose to install PAM yourself, I know I don’t).

Awkwardly, there is one component of NetBackup server that requires PAM: /usr/openv/netbackup/bin/bpjava-msvc. Unfortunately it’s a critical component. In order to administer a NetBackup server, you use the Java NetBackup Administration Console. When you connect to the NetBackup server with the console, the server authenticates the request by launching the authentication service from inetd; which is bpjava-msvc. Unfortunately for us, and for anyone else that doesn’t have PAM installed (even if they are running RHEL) bpjava-msvc is linked against the main PAM library libpam.so.0 and cannot launch without it:

I have spoken with Symantec about this and they clearly state that NetBackup is not dependant upon PAM, in its absence the Java client authentication service will happily authenticate against shadow, it just happens that the binary is linked to the pam library so needs it to be present to run. While there’s not much they are prepared to do about it right now, there is an “Idea” in the Symantec Connect portal’s Ideas section that requests the development team deal with this issue, and it is the right channel for communicating with product development. If you would like to see the issue dealt with, I heavily recommend you “Vote Up” the idea to bring it to their attention:
https://www-secure.symantec.com/connect/idea/bpjava-msvc-linked-against-pam-unnecessarily-making-it-dependency-entire-application

In the meantime, there is a really ugly workaround. It wants libpam.so.0? Well give it one; just not the rest of the PAM package. If you get the latest PAM source and compile it, you can copy the libpam.so.0 file into the NetBackup lib directory and voilá; everything works perfectly. While you could put it in /usr/lib64 or put it in /opt/pamlib and add /opt/pamlib to /etc/ld.so.conf, I recommend putting it in the NetBackup installation to save from unnecessary modification or pollution of your system. KISS. So put it in /usr/openv/netbackup/lib.

And that’s all there is to it. Now to add your tape/disk/removable devices and setup your backup policies.

Thanks to Symantec, migrating from my Solaris10-SPARC system is not an option. Even changing the hostname of the NetBackup server is not a supported operation and will break everything unless handled very delicately. Symantec say the only way they will help you is if you pay them a bucketload for Symantec Consultation Services. Well I can tell you now that just isn’t going to happen. In practice this means I am going to have to import all historic tapes into the new system (http://seer.entsupport.symantec.com/docs/327873.htm), but thankfully I have a Sun StorageTek C4 Robotic Tape Library that will hold 32 tapes at a time and I only need to import 60 or so tapes. It will probably take the best part of a week, but it’s acceptable.

YMMV

I can happily confirm that Veritas NetBackup’s RedHat2.6 client installer works absolutely perfectly for Slackware. So all of you who are stuck with Veritas NetBackup can rest easy.

Simply:

1. Run the installer
2. Delete the symlinks it creates in /etc/rc.d/rc{x}.d
3. Move /etc/init.d/nbclient to /etc/rc.d/rc.nbclient
4. chmod 755 /etc/rc.d/rc.nbclient
5. Add “/etc/rc.d/rc.nbclient start” to /etc/rc.d/rc.local
6. Add “/etc/rc.d/rc.nbclient stop” to /etc/rc.d/rc.local_shutdown
7. Configure client & server policies on the server.

Worth noting that I use this config for the Slackware clients at the moment:
/usr/openv/netbackup/bp.conf:

SERVER = [server-name]
CLIENT_NAME = [client-name]
USEMAIL = [e-mail address]
MEGABYTES_OF_MEMORY = 512
SERVER_SENDS_MAIL = YES
BUSY_FILE_PROCESSING
BUSY_FILE_DIRECTORY = /tmp
BUSY_FILE_ACTION = /* mail
BUSY_FILE_ACTION = /* repeat 2

Update: In Slackware64-13.0 you need the multilib stuff to get this working because it’s only released as x86 and IA64, not x86_64 – but there’s a gotcha. The installer provides its own copy of x86 tar which depends on x86 libacl.so.1 which even the multilib packages don’t provide. Initially I set about trying to put an x86 libacl.so.1 into the Slack64 box until I realised I was going about it the wrong way:

UPDATE 20100222:
NetBackup 7 has been released and I have it running as a server on Slackware64-13.1:
http://blog.tpa.me.uk/2010/02/22/netbackup-7-server-on-slackware64-13-1/