AI, Virtualization, Containers, Infrastructure, Linux
Due to the fact that /etc/modprobe.conf has been deprecated in RHEL 6, the process of bonding network interfaces has changed a bit.
Now instead of defining your bond in your /etc/modprobe.conf, you define it in /etc/modprobe.d/bonding.conf. No changes in syntax.
alias bond0 bonding
Then in your ifconfig-bond0 file, you define your bonding mode. Note the quotes, and the fact that I explicitly typed out the bonding mode.
BONDING_OPTS=”miimon=100 mode=active-backup”
The moral of the story today is beware of servers that have had a long history of changing networking configurations and network interfaces, as you never know what you are going to run into.
While working on a RHEL 5.4 box and attempting to make few minor changes to eth4 and up the interface, I was smacked with the error below.
e1000 device eth4 does not seem to be present, delaying initialization
This was very much suck.
Ifconfig-eth4 contained what you see below
DEVICE=eth4
BOOTPROTO=static
ONBOOT=yes
IPADDR=192.168.13.11
NETMASK=255.255.255.0
So first i hit up /etc/modprobe.conf and verifed that the correct driver module was being aliased for the interface.
alias eth4 e1000
Ok check. Modprobe.conf looks good.
Now re-initialize driver
modprobe eth100
Then onto check lspci to make sure that the driver initialized all the interfaces. Note that there are two different versions of the 82546EB driver, which tells me that the driver was only initialized for two of the interfaces initially.
2a:01.0 Ethernet controller: Intel Corporation 82546EB Gigabit Ethernet Controller (Copper) (rev 01)
2a:01.1 Ethernet controller: Intel Corporation 82546EB Gigabit Ethernet Controller (Copper) (rev 01)
2f:02.0 Ethernet controller: Intel Corporation 82546GB Gigabit Ethernet Controller (rev 03)
2f:02.1 Ethernet controller: Intel Corporation 82546GB Gigabit Ethernet Controller (rev 03)
Turned my attention back to the ifconfig-eth4 and the fact that there was not a MAC address defined in the config file. Grabbed the mac address out of /etc/sysconfig/hwconf and added it to the ifconfig-eth4 config file
class: NETWORK
bus: PCI
detached: 0
device: eth4
driver: e1000
desc: "Intel Corporation 82546GB Gigabit Ethernet Controller"
network.hwaddr: 00:04:23:cb:89:2e
vendorId: 8086
deviceId: 1079
subVendorId: 8086
subDeviceId: 1179
pciType: 1
pcidom: 0
pcibus: 2f
pcidev: 2
pcifn: 0
Then ifup works. Day is saved.
Do you know whats almost as delicious as a tasty waffle with pure AAA Vermont maple syrup smothered all over it. Neither do I.
However I did run across a very well written blog post on how to configure an NFS server on RHEL6.
Wierd, is it a NFS server or an NFS server, the latter sounds and looks better but i dont think its correct.
Anyway link below:
If you have ever evacuated disks in Veritas, every so often this will happen to hang. Usually you terminate your session or who knows what. Kinda like Joe Girardi's willingness to sacrifice outs for no good reason every time the Yankees hottest hitter is at the plate. It happens, you can't explain it, you move on. Back to technology – vxtask list shows no tasks, but you get errors trying to rerun the failed evac.
For example:
Plex %5 in volume rman is locked by another utility
Plex rman-01 in volume rman is locked by another utility
Subdisk rman_7_tmp-01 in plex rman-01 is locked by another utility
vxprint -hf is our best friend, as it shows you any flags that are set
v rman fsgen ENABLED 15625864960 - ACTIVE ATT1 –
pl %5 rman ENABLED 11719399168 - TEMPRM SDMVTMP –
sd rman_6-01 %5 ENABLED 1953232896 9766166272 - SDMVDST –
pl rman-01 rman ENABLED 15625864960 - ACTIVE SDMV1 –
sd rman_1-01 rman-01 ENABLED 1953234688 0 - - –
sd rman_2-01 rman-01 ENABLED 1953232896 1953234688 - - –
sd rman_3-01 rman-01 ENABLED 1953232896 3906467584 - - –
sd rman_4-01 rman-01 ENABLED 1953232896 5859700480 - - –
sd rman_5-01 rman-01 ENABLED 1953232896 7812933376 - - –
sd rman_6_tmp-01 rman-01 ENABLED 1953232896 9766166272 – SDMVSRC –
sd rman_7_tmp-01 rman-01 ENABLED 1953232896 11719399168 – - –
sd rman_8-01 rman-01 ENABLED 1953232896 13672632064 - - -
We can see that we have flags set on the temporary plex (from the failed evac), the subdisk for the temporary plex, the main plex, the subdisk in the main plex, as well as the volume itself. We need to clear flags to be able to finish re-start our evac. I will also cut the lines on the vxprint that don't change for the purpose of shortening this post.
vxmend -g rman_dg clear all rman %5
So we cleared the volume and temp plex flags, here's the vxprint -htf output afterwards
v rman fsgen ENABLED 15625864960 - ACTIVE - –
pl %5 rman ENABLED 11719399168 - TEMPRM - –
sd rman_6-01 %5 ENABLED 1953232896 9766166272 - SDMVDST –
pl rman-01 rman ENABLED 15625864960 - ACTIVE SDMV1 –
sd rman_6_tmp-01 rman-01 ENABLED 1953232896 9766166272 – SDMVSRC –
So now with the flags cleared we can remove the temporary plex
vxplex -g rman_dg -o rm dis %5
And once again our new vxprint -htf
v rman fsgen ENABLED 15625864960 - ACTIVE - –
pl rman-01 rman ENABLED 15625864960 - ACTIVE SDMV1 –
sd rman_6_tmp-01 rman-01 ENABLED 1953232896 9766166272 – SDMVSRC –
Great, now down to two flags, the one on the plex and the one on the source disk of our original evac. Clearing flags from subdisks is a lot trickier than clearing flags from volumes and plexes. Because the tutil0 flga is already set, we will need to force the clear. We clear by setting it to "".
vxedit -g rman_dg -f set tutil0="" rman_6_tmp-01
Once again, vxprint -htf
v rman fsgen ENABLED 15625864960 - ACTIVE - -
pl rman-01 rman ENABLED 15625864960 - ACTIVE SDMV1 –
sd rman_6_tmp-01 rman-01 ENABLED 1953232896 9766166272 – - –
And lastly, we clear the flag on the plex. Why in this order? Because I'm writing this up after I fixed my issues. In the interest of not editing vxprint outputs, it's like this. In retrospect, this could have been cleared with the first one we ran in the beginning.
vxmend -g rman_dg clear all rman rman-01
And finally, the way a vxprint -htf should look when all is healthy.
v rman fsgen ENABLED 15625864960 - ACTIVE - –
pl rman-01 rman ENABLED 15625864960 - ACTIVE - –
sd rman_1-01 rman-01 ENABLED 1953234688 0 - - –
sd rman_2-01 rman-01 ENABLED 1953232896 1953234688 - - –
sd rman_3-01 rman-01 ENABLED 1953232896 3906467584 - - –
sd rman_4-01 rman-01 ENABLED 1953232896 5859700480 - - –
sd rman_5-01 rman-01 ENABLED 1953232896 7812933376 - - –
sd rman_6_tmp-01 rman-01 ENABLED 1953232896 9766166272 – - –
sd rman_7_tmp-01 rman-01 ENABLED 1953232896 11719399168 – - –
sd rman_8-01 rman-01 ENABLED 1953232896 13672632064 - - –
At this point, feel free to proceed with your evac again. If you're wondering what the putil and tutil fields are, here is what I found courtesy of Symantec:
http://www.symantec.com/business/support/index?page=content&id=TECH15609
Guest Authored By: @momkvi
Honestly I cannot believe that i have not put this one up on Fatmin before. I have to add new disks to RHEL servers all the time and can never remember I can never remember this procedure. I end up hitting up google all the time.
Anyway the command below causes a rescan of the scsi bus which is what you need to do for your server to detect a newly added disk. I find myself doing this on virtual machines all the time. Oh and for the love of all that is holy, please create your new disk a logical volume, and dont be lazy and just format the disk and stick a filesystem on it.
In this example I am using host0 as the target, however this may differ on your box.
echo “- – -” > /sys/class/scsi_host/host0/scan
Fresh install of Fedora 15 on my home machine… feeling great, running great. Got myself a fast new SSD, and upgraded to a new quad core and 8gb of ram. Then i run into this.
[root@fedora15 ~]# service nscd start
Starting nscd (via systemctl): [ OK ]
Oh man whats this — systemctl. If this is anything like upstart I am going to be ill. Well guess what, it is. Even worse, its also kinda like svcadm in Solaris10.
"systemd is a replacement for the System V init daemon for Linux. It is intended to provide a better framework for expressing services' dependencies, allow more work to be done in parallel at system startup, and to reduce shell overhead"
Seriously, was there something wrong with systemV init scripts that i was not aware of. Looks like systemd is enabled by default in Fedora15,
Anway, the link directly below will take you to a nice cheatsheet for systemd commands. Looks like they are also mucking around with the sysivinit Runlevels. Scroll down for that little gem.
http://fedoraproject.org/wiki/SysVinit_to_Systemd_Cheatsheet
Below is also a FAQ on systemd
http://www.freedesktop.org/wiki/Software/systemd/FrequentlyAskedQuestions
And a bit more insight into what it is and where it comes from
Finally the blog post from the developer announcing systemd. Apparently from last year.
http://0pointer.de/blog/projects/systemd.html
Wondering now when we are going to see this in RHEL and if and when systemVinit will be completely deprecated.
Got a heads up from a DBA today stating that one of the their database servers was running hot from an i/o perspective. So in order to troubleshoot jumped on the server and ran an iostat
>iostat -d -x 5 3
Based on the output below i was able to determine which disks where the most utilized by looking at the last column which is %utilized. Two other columns to take note of are await and svctime,where await time is the average response time in ms for an i/o request to the device, including any time spent waiting in a queue. Svctime is the average time it took to service a request after it was sent to the disk and out of the queue. In this case service times are low so i can pretty much rule out SAN issues.
VxVM65519 0.00 0.00 230.80 20.80 5753.60 556.80 25.08 1.77 6.94 3.50 99.08
VxVM65516 0.00 0.00 257.60 21.80 6022.40 467.20 23.23 1.68 6.02 3.01 93.22
VxVM65515 0.00 0.00 265.80 18.80 6563.20 364.80 24.34 1.42 4.89 2.69 94.58
VxVM65513 0.00 0.00 233.20 24.00 6032.00 969.60 27.22 1.41 5.47 2.99 88.92
VxVM65493 0.00 0.00 308.80 21.00 7590.40 944.00 25.88 1.74 5.27 2.52 98.14
VxVM65492 0.00 0.00 262.80 20.40 6502.40 716.80 25.49 1.76 6.10 2.94 99.22
In this case svctimes are ok so i know that the disks are performing well, however i do know that the hot disks are under Veritas control, so i take a quick look at the disk groups on the box
>vxdg list
Then do i vxstat on each disk group individually to look for the ones with high number or reads and rights. Then i can advise the dbas which specific database is causing the load on the box.
>vxstat -g example_dg -i 1