Aaron Toponce

At work, we use Symantec NetBackup for all our full and incremental backups to tape. With currently about 100 servers in operation, and roughly 300+ virtual development environments on the horizon using RHEV, NetBackup and Bare Metal Restore, also from Symantec, will be crucial in our operations.

Recently, we had some virtual servers take a dive, because the disk array they were on took a dive, and with it, all the storage that these VMs were relying on for their OS and other needs. So, we've been building these machines back up from backup on a new NetApp SAN. Everything has been working well with Bare Metal Restore for Windows VMs, but we were struggling getting it to work with RHEL. So, I got the idea to boot up a Fedora 12 Live CD, install the NetBackup client in the live environment, and do a restore from there, then reboot into the newly built machine.

Works like a charm! So, I documented the steps necessary to make it work. Realize, that this was documented for my place of employment. Of course, I've completely changed the hostnames, IP addresses, and other details that would be specific to work. However, if you deploy this in your environment, then you'll need to make the necessary architectural changes that fit your needs. Also, this should work with Fedora 13, the latest release at the time of this writing, but Fedora 12 works well for us, and because it's used just for this purpose, we see no pressing need to use the latest. Also, Debian or Ubuntu, or some other GNU/Linux based operating system might work as well, but NetBackup doesn't support these operating system necessarily, and it is expecting the filesystem layout that is common with Red Hat based operating systems. Regardless, Fedora 12 is being used as a tool here, not because of any loyalty or fanboyism.

So, here's the tutorial:

Fedora can be used as a temporary live environment for NetBackup when doing restores to RHEL machines. Just boot from the Fedora ISO or CD, and make sure you have configured networking appropriately, and that you can get out to the Internet. After that, follow the instructions below. This does assume that you have a base, bare-bones, RHEL install that matches the partitioning or volume layout of the previous install that you're restoring. This must be completed first, because at the end of the tutorial, you will be mounting these devices, and restoring only the files, to these mount points.

First, the liveuser can get root access without any password. However, for this tutorial, you will need to set a password for root, so go ahead and do that:

[liveuser@localhost ~]$ su -
[root@localhost ~]# passwd
Changing password for user root.
New password:
Retype new password:
passwd: all authentication tokens updated successfully.
[root@localhost ~]# 

At this point, we need to setup the hostname and networking correctly. We have set aside the 'netbackup.example.com' domain name for the Linux restore system in our DNS environment, and the IP address that has been set aside is '10.19.84.254'. In the terminal as root, set the hostname:

[root@localhost ~]# hostname netbackup.example.com
[root@localhost ~]# export HOSTNAME="netbackup.example.com"

Now get networking configured with NetworkManager. In the status bar of the desktop, there is an icon that looks like two computers. Right click this, and click “Edit Connections…”. Click “Auto eth0”, click the “Edit…” button, click the “IPv4 Settings” tab, and set the parameters as necessary:

Method: Manual
Address: 10.19.84.254
Netmask: 255.255.255.0
Gateway: 10.19.1.254
DNS servers: 10.19.2.192, 10.19.3.192, 10.19.4.192
Search domains: example.com

Click “Apply…”, and enter the password for root, then close the “Network Connections” widget. The icon looking like two computers should have a red “X” next to it now. This means it's not online. Left-click the icon, and click “Auto eth0” to get it back online. Back in your terminal, you should be able to verify that the networking has been set correctly:

[root@localhost ~]# ip a show eth0
2: eth0:  mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:1a:4a:06:14:24 brd ff:ff:ff:ff:ff:ff
    inet 10.19.84.254/24 brd 10.5.31.255 scope global eth0
    inet6 fe80::21a:4aff:fe06:1424/64 scope link
        valid_lft forever preferred_lft forever
[root@localhost ~]# ping -c 4 mediaserver.example.com
PING mediaserver.example.com (10.19.84.60) 56(84) bytes of data
64 bytes from mediaserver.example.com: icmp_seq=1 ttl=64 time=0.942 ms
64 bytes from mediaserver.example.com: icmp_seq=2 ttl=64 time=0.230 ms
64 bytes from mediaserver.example.com: icmp_seq=3 ttl=64 time=0.193 ms
64 bytes from mediaserver.example.com: icmp_seq=4 ttl=64 time=0.212 ms

--- mediaserver.example.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3002ms
rrt min/avg/max/mdev = 0.193/0.394/0.942/0.316 ms

In order for the NetBackup client to communicate with the media server (mediaserver.example.com - 10.19.84.60), we need to make a modification to our /etc/hosts file, and add it's IP address and hostname:

[root@localhost ~]# echo "10.19.84.60      mediaserver.example.com mediaserver" >> /etc/hosts

With networking correctly set, we need to start up the SSH server:

[root@localhost ~]# service sshd start
Generating SSH1 RSA host key:                              [  OK  ]
Generating SSH2 RSA host key:                              [  OK  ]
Generating SSH2 DSA host key:                              [  OK  ]
Starting sshd:                                             [  OK  ]

Because this is a temporary live environment, we're not interested in SELinux nor any firewall. So, let's get those disabled first:

[root@localhost ~]# getenforce
Enforcing
[root@localhost ~]# setenforce 0
[root@localhost ~]# getenforce
Permissive
[root@localhost ~]# service iptables stop
iptables: Flushing firewall rules:                         [  OK  ]
iptables: Setting chains to policy ACCEPT: filter          [  OK  ]
iptables: Unloading modules                                [  OK  ]

At this point, you should verify that you can SSH to the Fedora live environment from another host. It should work. However, we still have more work to do. Before we can get the NetBackup client installed, we need to install a few packages. So, in your terminal, type the following:

[root@localhost ~]# yum install glibc-2.11-2.i686 libacl-2.2.47-5.fc12.i686 libstdc++-4.4.2-7.fc12.i686

This will download about 30MB of packages and metadata necessary for installation. Depending on your Internet connection, this might take a while. If you don't want to wait, you could change the repo files in /etc/yum.repos.d/, and use a faster mirror. Or, you can wait. After the packages are installed, and everything configured, we should be able to install the NetBackup client. To do this, we need to SSH to mediaserver.example.com, and send the client over. You can do this from the terminal you've already been working in:

[root@localhost ~]# ssh mediaserver
The authenticity of host 'mediaserver (10.19.84.60)' can't be established.
RSA key fingerprint is 14:d3:50:f1:27:d5:14:ab:c2:ca:51:fa:7e:5a:98:4c.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'mediaserver,10.19.84.60' (RSA) to the list of known hosts.
root@mediaserver's password:
[root@mediaserver ~]# 

Now transfer over the client. First, mediaserver.example.com will likely have the wrong SSH key for this box on file, as you might have done this more than once, and every time you boot into the Fedora 12 Live CD, and start up the SSH server, a different key will be generated. You will need to delete it:

[root@mediaserver ~]# cd /usr/openv/netbackup/client/Linux/RedHat2.6/
[root@mediaserver RedHat2.6]# ./sftp_to_client netbackup.example.com root
Connecting to 10.19.84.18...
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
89:db:1f:12:f9:8f:76:38:63:6e:54:75:7a:43:ed:9e.
Please contact your system administrator.
Add correct host key in /root/.ssh/known_hosts to get rid of this message.
Offending key in /root/.ssh/known_hosts:4
RSA host key for netbackup.example.com has changed and you have requested strict checking.
Host key verification failed.
Couldn't read packet: Connection reset by peer

sftp connection to netbackup.example.com failed.

Notice, that it says the offending key is on line 4 of /root/.ssh/known_hosts. So, I'll delete line 4 from that file:

[root@mediaserver RedHat2.6]# sed -i 4d /root/.ssh/known_hosts

Now, try again:

[root@mediaserver RedHat2.6]# ./sftp_to_client netbackup.example.com root
Connecting to netbackup.example.com...
The authenticity of host 'netbackup.example.com (10.19.84.254)' can't be established.
RSA key fingerprint is 4d:74:e6:66:ed:d9:bc:dc:3a:71:41:51:18:58:da:e9.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.19.84.254' (RSA) to the list of known hosts.
root@netbackup.example.com's password: 

sftp completed successfully.

The root user on netbackup.example.com must now execute the command
"sh /tmp/bp.22802/client_config [-L]".  The optional argument, "-L",
is used to avoid modification of the client's current bp.conf file.
[root@mediaserver RedHat2.6]# exit
[root@localhost ~]#

Now that we are back in the live environment, we can execute the command as given us from mediaserver.example.com. Of course, the name of the file transferred over might be different. First, let's get xinetd running:

[root@localhost ~]# service xinetd start
Starting xinetd:                                           [  OK  ]
[root@localhost ~]# sh /tmp/bp.22802/client_config

It will spit out a bunch of files and directories that it's installing for the client, and will reload the xinetd server. All that is left, is to start the NetBackup client, and we're finished!

[root@localhost ~]# service nbclient start
NetBackup SAN Client Fibre Transport daemon started.

Verify that the default NetBackup ports 13722, 13724, 13782 and 13783 are bound to the system:

[root@localhost ~]# lsof -Pan -i tcp | grep 137
xinetd   3261 root    5u  IPv6  22914      0t0  TCP *:13782 (LISTEN)
xinetd   3261 root    6u  IPv6  22915      0t0  TCP *:13722 (LISTEN)
xinetd   3261 root    8u  IPv6  22916      0t0  TCP *:13724 (LISTEN)
xinetd   3261 root    9u  IPv6  22917      0t0  TCP *:13783 (LISTEN)

At this point, you should be able to login to the NetBackup Administrator Console, find the netbackup.example.com client, and verify that they can talk. The only thing that you should be aware of, is that you need to mount all logical volumes and partitions to their respective mount points BEFORE the restore! This is important! When in the NetBackup Administrator Console, and doing the restore, you restore the files to a specific mount point. In our case, it's going to be '/'. This means, that if the host has separate /var, /u01, /, /home and other mount points, they all need to be mounted to their respective mount points BEFORE doing the restore!

[root@localhost ~]# mount /dev/work/root /mnt
[root@localhost ~]# mount /dev/work/var /mnt/var
[root@localhost ~]# mount /dev/work/u01 /mnt/u01
etc

CONGRATULATIONS! At this point, you're ready to start restoring the data using NetBackup in the Fedora 12 live environment. Just remember, that when you're in the NetBackup Admin Console, you need to restore the data to netbackup.example.com, and the directory path you need to use should be /mnt, not /, obviously.

I hope someone on the remote corners of the Internet finds this tutorial helpful. If so, let me know if the comments. If there are any spelling errors or I've missed something giving away detail of my works, also let me know so I can make the corrections. Thanks!