The other day a co-worker was doing his rounds in the data center, sent the team an email with (photos attached) indicating amber lights on one of the shelves of our Netapp Filer. Amber lights on a shelf of disk on your SAN usually isn’t a good thing so I jumped onto OnCommand System Manager to begin the troubleshooting.It didn’t take me very long to figure out the problem because the first thing I see after I am logged into OnCommand was a warning that I had “unowned disks”.
The aha moment hit me, just the day before we had 2 disks fail and were replaced via the configured Autosupport. Gosh, I do like the autosupport. Having a disk go bad overnight then getting a replacement before you even walk through the door is very convenient.
When these particular disks were replaced by another co-worker the day before, he had removed the bad disks and inserted the replacement disks. Since our filers are setup with software disk ownership , with disk auto assign disabled, the disks were not assigned automatically to an owner. Ownership must be assigned for disk(s) before the filer can use them otherwise they are useless and flash amber lights at you.
To assign ownership to the disk(s) SSH into your Filer and do the following :
- You will need to locate the disk(s) that don’t have any owners. Type the following command
disk show -n
- Once you have the disk name of the unowned disk you assign ownership with this command:
disk assign <disk_name>
disk assign 0b.16
for multiple disks < disk assign 0b.43 0b.24 0b.27
Assign ownership for all unowned disks at once < disk assign all
- Run disk show -v to verify the disk assignments
So there you go, a pretty simple fix. Next time you replace a failed drive don’t forget to give it an owner!
Netapp filers can be accessed and managed many ways, including using Putty to SSH into the filer itself. In addition to FilerView, there is also another web based tool called Netapp OnCommand System Manager that is GUI based which gives a very nice graphical performance chart detailing how HOT your filers are running. The OnCommand tool is great for everyday management of the filers but sometimes you will need to access the filers via Putty to run more advanced functions , ie. killing a NDMP session that is hung.
We had an interesting issue today while trying to access one of our Netapp filers using Putty. Every time we would we try try to log into the filer with a Putty session we would get an access denied or the Putty session would simply close. What was odd was that it didn’t happen for all of the us Storage Engineers. Thinking that maybe are accounts are locked or maybe the access got removed I started the OnCommand session and attempted to log into the filers.
Not a single hiccup. Logged in right away on every single filer we have. hmmm….so I can log in with my credentials using OnCommand but when using a Putty session I can’t. Yet, another storage engineer can login to both and we all have the same permissions. All filers were checked for locked accounts including Active Directory, nothing was locked.
After some more head scratching one of the other Storage Engineers stumbled upon a setting within the OnCommand System Manager setting that was caching our passwords. Once the tick box to cache passwords was cleared we were able to successfully log onto the filers.
To remove the cache passwords in OnCommand :
- Run OnCommand System Manager and log onto any filer
- In the top left hand corner select to Tools
- Select Options
4. Clear the Enable Cache Passwords tick box
- Select Clear Existing Passwords
Select Save and Close
Once the settings were changed we were both able to Putty to the filers. Gotta Love the gotchas of cached passwords.