Tips & Tricks

Setting up Data Protection on Tintri T540 – Replicate and Restore

If you read my earlier post about Tintri’s VMstore T540 you know that setting up data protection on the T540 is very simple. There is not much to it really just a few simple clicks and you’re replicating between applicances. Once you set the Replication settings ( assigning the replicating partner and password) and add the purchased license on the appliance you are ready to configure the vms for protection.

Follow the below settings to setup replication of a vm and restore it on another T540. You’ll find that it’s very simple to setup and takes just a few minutes.

Add Replication settings to appliance

Add Replication License

Configuring VMs for Snapshot & Replication to another T540

Login to your T540 using the url of the appliance http://x.x.x.x/
Click on “Search VM” link (top right)
A listing of all the VMs will be shown
Find the Virtual Machine you wish to protect and right click and select Protect.
Select the snapshot schedule, retention period, and if you want the snapshot to be replicated and the alert threshold. When you are completed Click on Protect.
To verify the replication state, on the Virtual Machine list right click on the header bar and select Data Protection. This will refresh the Protection tab showing the replication state, schedule and retention.

Restoring a Replicated VM on another T540

Login to T540 hosting snapshot using the url of appliance http://x.x.x.x/
Click on “Search VM” top right
A listing of all the VMs will be shown. Click on Snapshots and this will lists all snapshots held on the appliance.
Locate the snapshot you want to recover. Right click and select Clone.
Create new Virtual Machine window will open. Fill out the required fields and click Clone. The window appear at the bottom right informing you that the Virtual Machine is being added to the inventory.
Sign into vCenter and find the Virtual Machine that you just cloned complete any needed configuration changes and power on.

Don’t Forget to Assign Ownership to Disks on your Netapp Filers

The other day a co-worker was doing his rounds in the data center, sent the team an email with (photos attached) indicating amber lights on one of the shelves of our Netapp Filer. Amber lights on a shelf of disk on your SAN usually isn’t a good thing so I jumped onto OnCommand System Manager to begin the troubleshooting.It didn’t take me very long to figure out the problem because the first thing I see after I am logged into OnCommand was a warning that I had “unowned disks”.

The aha moment hit me, just the day before we had 2 disks fail and were replaced via the configured Autosupport. Gosh, I do like the autosupport. Having a disk go bad overnight then getting a replacement before you even walk through the door is very convenient.

When these particular disks were replaced by another co-worker the day before, he had removed the bad disks and inserted the replacement disks. Since our filers are setup with software disk ownership , with disk auto assign disabled, the disks were not assigned automatically to an owner. Ownership must be assigned for disk(s) before the filer can use them otherwise they are useless and flash amber lights at you.

To assign ownership to the disk(s) SSH into your Filer and do the following :

You will need to locate the disk(s) that don’t have any owners. Type the following command
disk show -n
Once you have the disk name of the unowned disk you assign ownership with this command:
disk assign <disk_name>
disk assign 0b.16
for multiple disks < disk assign 0b.43 0b.24 0b.27
or
Assign ownership for all unowned disks at once < disk assign all
Run disk show -v to verify the disk assignments

So there you go, a pretty simple fix. Next time you replace a failed drive don’t forget to give it an owner!

Remove Hidden Devices From Servers

One of the things I do after a successful P2V of a server is do some cleanup on the newly virtualized server. Cleaning up involves uninstalling any software , such as Broadcom and Dell Open Manage software, that is no longer needed for the server. I also perform clean up of the any old or removed devices through the servers Device Manager that may still be lingering around. Removing these hidden or ghost devices can prevent or fix any issues you may have with the vm.

To remove the hidden devices you will need to go to Device Manager and tell it to Show Hidden Devices, the devices will be displayed in a lighter shade than the installed devices. However, after performing many P2Vs , I found out that Device Manager doesn’t display all the devices that server may have had at one point.

Fig1.Hidden

Fig2.Hidden

One would think by selecting this view everything would magically appear based on the fact that we just told Device Manager to “show hidden devices”. Since we are dealing with Windows and something that would be logical isn’t, we can the modify the registry to see these “hidden” devices . Once the registry value has been added ,reboot the server then you will be able to see all those hidden devices that were never uninstalled and you can properly uninstall them.

Add the following Registry Settings

Registry Settings
System Key: [HKEY_LOCAL_MACHINE\SYSTE\CurrentControlSet\Control\Session Manager\Environment]
Value Name: DEVMGR_SHOW_NONPRESENT_DEVICES
Data Type: REG_SZ (String Value)
Value Data: (1 = show all hidden devices)

Fig3. Hidden

After Rebooting the server go back into Device Manager < Select Show Hidden Devices again and you will now see the hidden devices shaded lighter.

Fig4. Hidden

To remove a device , right click on the device and select uninstall.

Fig5. Hidden

You may be prompted to confirm that you want to remove the device , select OK and the device will be removed.

Fig6. Hidden

You can remove hidden devices on VMs or even physical servers that has been running for a while as part of overall server maintenance plan to keep your servers running in tip top shape.

TechEd 2014 – It’s more than KoolAid and Swag

IT conferences are a place where us nerds flock to a designated destination to discuss all things IT related and get your yearly supply of vendor t-shirts. Microsoft’s TechEd conference is no different and this year it’s held on May 12-15 in Houston, TX. Tech conferences are more than the KoolAid and who has the best swag, it’s also about the parties/events and the people you meet there. The networking that one can do at a tech conference is amazing; you have vendors and IT professionals all in one location at your disposal. Making those connections can mean new business deals or even a job, you never know the sky’s the limit. Not only are you building your IT professional network but attending social events after hours keeps you from watching re-runs on TV in hotel room. Who really wants to do that? Not I.

If you are new to TechEd or don’t know where to start with all this social activity I’ve made a list of some events that you may want to check out. You never know you may even run into me at one these events…

Sunday Night 5/11/2014

The Krewe Meet and Greet at House of Blues – 7:30pm – 10:30pm

Kick off TechEd by meeting members of the world famous #TheKrewe- #TheKrewe is a community organized professional networking group. Their mission is to build a dynamic networking community through various activities and gatherings that extend beyond TechEd. The event is on a wait list but email meetngreet@iamkrewe.org to try to get yourself on the wait list.

Monday Night 5/12/2014

TechEd Expo Welcome Reception – 6:00pm – 8:00pm

Tuesday Night 5/13/2014

Petri IT Author Meet and Greet – 5:00pm- 7:00pm

Come meet the authors of Petri IT knowledgebase (including myself!) sponsored by Veeam and Proximal Data.
The following Petri authors, contributors, and editors should be in attendance:

Aidan Finn – @joe_elway
Damian Flynn – @Damian_Flynn
Phoummala Schmitt – @PhoummalaSchmit
Jeff James – @JeffJames3
Daniel Petri – @p3daniel
Peter De Tender – @pdtit
J. Peter Bruzzese – @jpbruzzese
Mary Branscombe – @marypcbuk
Simon Bisson – @sbisson
Jonathan Hassell – @jghassell
Jeffrey Hicks – @jeffhicks

Space will be limited – RSVP ticket by clicking on the following link and registering for the event http://www.eventbrite.com/e/petri-it-knowledgebase-author-meet-and-greet-at-microsoft-teched-2014-tickets-11225258049

Join NetApp for craft beer, food and fun at Saint Arnold Brewing Company- 4:30–7:30 p.m

Stop by at the Netapp Booth with your TechEd Badge before Tuesday for an invite and party badge.Transportation to and from will be provided. Pick up will begin at 4:15 p.m. and runs through

5:30 p.m. outside of Hall F at the George R Brown Convention Center. Just look for the

NetApp logos.

Veeam TechEd Party at House of Blues – 8:00pm – 11:00pm

To register for the Party Apply stop by at the Veeam booth or pre-register here: http://go.veeam.com/teched_party_2014.html

CommVault TechEd Jam Session at House of Blues- 9:00pm – 1:00am

Come hang out with your fellow attendees and see who has some musical talent. If you’re not into singing or jamming on stage just sit back relax and enjoy a beer, you might even see me swinging by.

Wednesday night 5/14/2014

TechExpo Happy Hour and Best of TechEd North America 2014 Awards – 5:00pm – 6:00pm

7th Annual UC Roundtable – 6:00pm – 8:00pm

The purpose of the UC Roundtable is to gather Exchange and Lync admins, MCMs, MVPs, Exchange product group members, architects, and experts for a free-flowing discussion about issues, questions, and experiences related to Exchange, Office 365, and Lync Server. If you work with Exchange, Office 365, or Lync you need to be at this party. Come join other Exchange and Lync admins, MCMs, MVPs, Exchange product group members, architects, and experts to discussion about issues, questions, and experiences.

Please RSVP to jeff@expta.com for event details and location

Scheduled Maintenance Party hosted by ENow – 9:00pm – 12:00am

The awesome folks at ENow are bringing all of us Exchange, Lync, and SharePoint admins much needed “Scheduled Maintenance”. This is a party you don’t want to miss. It will be EPIC!

You can also stop by the ENow booth for more details.

Go to http://schedulemymaintenance.com to request a pass.

Meet the DevExpress Team at TechEd and Lucky Strike Houston! – 7:00pm -11:00pm

Come meet the DevExpress Team and enjoy a night of cocktails, appetizers and bowling.

RSVP as soon as possible @ rsvp@devexpress.com

Thursday night 5/15/2014

TechEd 2014 Closing Party at Minute Maid Park – 7:30pm 11:00pm

Close out TechEd by enjoying an evening at Minute Maid Park where attendees have exclusive access to the entire park. There will be lots of food, drinks, music, games, and activities. You must have your wrist bracelet to attend party. Guests of attendees can purchase tickets for the party , see information/registration desk at convention hall.

If you need a daytime guide for TechEd, check out some of these sites:

SnapManager for Exchange fails to run scheduled snaps after running an upgrade to 6.0.4

Sometimes fixes & patches introduce another set of issues that will give way to another set of new patches and fixes.

In our case, it was our upgrade to SnapManager for Exchange( SME) 6.0.4 which had fixes to some bugs we were facing. Everything seemed to go real well, all the upgrades on the Exchange 2010 DAG member servers didn’t hiccup one bit. This was too good to be true, an upgrade of SME and no issues so far. I had my fingers crossed and was hoping for the best, maybe luck would be in our corner.

No Joy…

After completing the upgrade on all servers I needed to run a test of some exchange snaps. Got to make sure it works right? I first started out running manual snaps on all the databases on each node. Those worked great, No Problems.

So onward to the next test which was to kick off a scheduled snap of the DAG databases. After kicking off a scheduled snap through task scheduler the snaps failed to run. After some digging around and a few more tests, my co-worker discovered that there is bug when you upgrade to SME 6.0.4 which causes scheduled snaps to fail.

According to Netapp’s KB 649767 article it has to do the value “0” is not selectable in the “retain up-to-the-minute restorability” option in the GUI of this release like it was in previous releases. When running the snaps through the GUI of SME 6.0.4 , you can manually enter the value “0” and the run the job immediately, backups will work. The issue occurs when SME creates a scheduled job; it creates the job with wrong parameter , it be should be NoUtmRestore if you don’t want to retain any transaction logs.

http://support.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=649767

Getting Backups to work again…

To get scheduled backups to work again you will need to do one of 2 things:

Change the -RetainUTMDays and -RetainUTMBackups from something other than “0”. Changing the value to something other than “0” will retain your transaction logs for the specified value
If you don’t want to keep any transaction logs, manually modify the scheduled job and remove the -RetainUTMDays or -RetainUTMBackups parameters then replace with NoUtmRestore.
- If you are running a DAG remember you will need to modify the scheduled job across all DAG members that have the scheduled job.

Using Custom Filters For Your Exchange Dynamic Distribution Groups

Dynamic Distribution groups are Distribution groups that dynamically add members into the group based on a certain set of filters and conditions, when an email is sent to the group. These are great for mass mailing a group of users that change can often and managing the group manually would be difficult to maintain.

Exchange offers 2 ways of creating these groups; you can use the EMC/ EAC or PowerShell. I have found that the majority of cases for Dynamic Distribution can be created using the EMC/EAC, which offers the following set of pre-canned filters and conditional.

IncludedRecipients
ConditionalCompany
ConditionalDepartment
ConditionalStateOrProvince
ConditionalCustomAttribute( 1–15)

There are times that this pre-canned list just doesn’t fit the bill. Let’s say you need a Dynamic Group that filters on users from a certain country or even a particular job title? PowerShell to the rescue!

PowerShell offers the pre-canned filters as well as any of the account attributes that a user account would have, giving you a lot more freedom to create some customized Dynamic Distribution Group. Please note that you cannot combine pre-canned conditional filters and custom Recipient Filters in the same query.

For example, to create a Dynamic group for mailbox users only in a particular country and company, let’s say the US, use the following cmdlet:

New-DynamicDistributionGroup -Name "TestGroup" -Alias "TestGroup" -OrganizationalUnit "your/OU"-RecipientFilter {(RecipientType –eq  “UserMailbox”) -and (CountryOrRegion –eq “United States”) -and (Company –eq “mycompany”)}

If you have an existing group that you just need to modify to become custom use the Set-DynamicDistributionGroup cmdlet:

Set-DynamicDistributionGroup -Identity "TestGroup" -RecipientFilter {(RecipientType –eq  “UserMailbox”) -and (CountryOrRegion –eq “United States”) -and (Company –eq “mycompany”)}

Note that when creating your Dynamic Distribution Group using PowerShell you cannot combine pre-canned conditional filters and custom Recipient Filters. A list of all the available filterable properties parameters can be found on TechNet’s site.

http://technet.microsoft.com/en-us/library/bb738157(v=exchg.150).aspx

Exchange 2010 and Active Directory Operation Failed on DC errors

An annoying problem that I have seen since we upgraded to Exchange 2010 is when in the Exchange Management Console (EMC) , you are not able to perform certain tasks because a DC could not be contacted .The domain controller in the error is usually one that has been demoted from your environment but sometimes not. The issues can also occur after recent changes to a DC, which causes the EMC to lose contact with the Domain Controller

When this particular scenario was first noticed , it puzzled us because the DC in question was still running and Exchange was able to discover it. We did all the typical AD and exchange troubleshooting steps, checked permissions, AD replication, etc., but none these steps fixed the issue, the tech was still not able to create accounts. After some more digging around we later found out some FSMO roles were removed from that DC. Aha! A major change to the DC.

Common error messages may contain “Active Directory operation failed on Dcxxxx” or “ LDAP server was unavailable”. When the problem occurs you are not able to perform certain actions in the EMC, such as creating accounts, mailbox moves, basically any operation that requires contact with the DC.

An example of an error is shown below:

So what’s the problem you ask?

The problem is a result of the Exchange Management Console caching the domain controller details in the MMC temp files. It caches the data but it’s not smart enough to update the data or locate another DC. To fix the issue you have to remove the MMC cache file from the users profile.

Use the following steps to clear the EMC MMC cache file:

1. Close the EMC if you have it open
2. Go to the User’s profile directory and delete the Exchange Management Console file.
3. File location can be found here:

C:\users\<specific user>\AppData\Roaming\Microsoft\MMC\Exchange Management Console

4. Reopen the EMC

See Microsoft KB article http://support.microsoft.com/kb/2019500

Change a Windows Server 2012 Product key?

For those that are just getting started with Windows Server 2012 you may have noticed that you don’t get the option to change the server’s product key on the activation page. This can be a problem if you need to switch to MAK or KMS keys and reactive the server. No worries, the solution is pretty simple. You can use one of the following 2 options:

Option #1 –

1. Open a Command promtp and run the following :

slmgr.vbs /ipk <put your Product key here>

Option #2 –

1. On Windows server 2012 -( with your mouse point to the lower-right hand corner of the screen) click Search.

2. Type Slui.exe 0x3

3. When the activation screen pops up type in your new Product Key and Activate.

Message restrictions on your Exchange groups preventing emails to be received?

A common problem I get support tickets on is distribution groups not receiving emails from non-Exchange systems. There can be various reason for this, but the first thing I do is check the mail flow settings for the group to determine if the group is restricted to only authenticated users.

When groups are created in Exchange 2010, the default settings are applied and off you go, ticket closed. What is often overlooked are the mail flow settings for the group. The mail flow settings control who can send to the group, which senders to reject messages from and require that all senders are authenticated. Typically this not an issue if you’re sending emails to the group in Outlook, but if you want to send emails to this group from outside of the Exchange system, you will want to make sure that this is not enabled. When the “require that all senders are authenticated” is enabled it prevents any users that don’t have valid logon credentials in your organization from sending emails to this group. If you have servers in your environment that need to send emails to an Exchange group, you will want to disable this setting since that server is not authenticating. If the groups were created prior to Exchange 2010, this setting was not enabled by default therefore there were restrictions.

You can disable the setting by 2 ways, through Powershell or the Exchange Management Console (EMC).

To disable the setting in the EMC:

Go to the group properties in the EMC > Mail flow settings > clear the checkbox on “Require that all senders are authenticated”

To disable the setting in Powershell:

Set-DistributionGroup -Identity <DistributionGroupIdParameter> -RequireSenderAuthenticationEnabled $false

Expanding System Drive of a running W2K3 VM using EXTPART

At some point in time a VM will run low on disk space and you’ll need to add more space to it. For VMs running Windows server 2008 this isn’t an issue but anything older, well , you had to use tools such as DiskPart to get the job done. Let’s say the system drive was running low, for W2K3 servers, this was a little bit harder to expand because you needed to shutdown the VM. Depending on what you were running on that VM getting that shutdown approval could take an act of congress to pass. Expanding the system drive on a running W2K3 VM is not impossible though, for I have the solution that could solve those woes.

A few years ago when my company was just getting started on our virtualization infrastructure (VMware of course) I became familiar with a nice little tool that would expand drives on the fly for VM’s running W2K3 or older. This was pretty important considering over 80% of our servers at the time were still on W2K3. What’s this magical tool that can save us admins from having to shutdown a VM because the users hoarded all the space?

Dell’s EXTPART. I will tell you that there is no guarantees with this and use at your risk. Since I’m a daring person , well honestly, just don’t want to go asking to shutdown the server, I’ve used this tool. I have used it quite often too. I’m still alive, the servers survived, and half the time they didn’t even know they were low on space because I proactively fixed the problem.

Now, this isn’t the magical tool that fixes everything because it does have some caveats. For instance, you can only expand system drives on while running if the VM was not P2V’d. If the VM’s were built from scratch or from a template this tool works great but if your VM was P2V’d you need to reboot into safe mode to expand the system drive. Expanding non system drives while running is perfectly fine on P2V’d VMs using EXTPART.

To use this EXTPART do the following:

To use EXTPART to expand a drive you will first need to download the tool and save it to a location on the VM. The tool can be found here
In vSphere, go to the settings of your VM and increase the Hard disk that you want to expand. If it’s the system drive then most likely you will want Hard disk1.

Now go to your VM, open a Command Prompt and navigate to the directory where you saved the download.
Once you are in the file directory of EXTPART, run the extpart.exe file
You will be prompted to type in the Volume or Drive that you want to expand, type in C:
After putting the volume it will display the details of the drive, such as volume and partition size, and ask how much you want to expand the drive. At this point enter in the size (MB) that you want to expand the drive by. This is the amt you are increasing the drive with not the total size of the drive.

Once complete press enter and the size of the increased volume will be displayed.
If this fails with an error that “the disk is not accessible” then most likely the VM was P2V’d and you will need to reboot the VM into safe mode. Once booted into safe mode start at step 3 again.

So there you have it, an expanded drive on a running W2K3 VM. No time downtime and you look like a hero for expanding the drive. #WINWIN

Exchange Goddess

I'm here for the refreshments. About cows, goats, and surviving in IT as a woman