Cleaning up disk space on a SBS 2008 (or 2011) Server

Cleaning up disk space on a SBS 2008 (or 2011) Server

This is a follow up posting to my popular blog entry on reducing the Sharepoint DB log files, posted back in 2010.

When we deployed SBS 2008 servers back in 2008 and the following years, Microsoft told us to size the system (C:) drives to 60GB. All deployed there would typically be about 20GB of disk space left. However the WinSXS folders have over the years grown and now occupy at least 15GB of disk space, leaving apprx 5GB to play with. So tuning and conserving disk space on a SBS 2008 server has become a (not so) favourite maintenance task on our early SBS servers.

Here are a few tips and links to clean-up tasks we perform.

The SBS team has released their own blog post that is fairly comprehensive, see 'Recovering disk space on the c drive in SBS 2008'.

If your site uses Sharepoint extensively, make sure you check out 'Reduce a huge Sharepoint MSSCrawlURLLog table in the WSS_Search database',  by Charles De Guchteneere.

Next up is the SBS Monitoring database, which is again a fix by the SBS team, see '
How to Recreate the SBSMonitoring Database'.

While there, stop the Windows Small Business service, then navigate to the
C:\Program Files\Windows Small Business Server\Logs\MonitoringServiceLogs
folder. You will note two large log files:
ConfigurationHelper_Policy.log
DataServiceComponents.log
Rename them to .bak and create two empty log files with the same name. Once the service has been resatrted and you don't think you need the contents of the files need to be checked, you delete them. Bang, often 1GB of disk space freed!

A request to allocate an ephemeral port number from the global UDP port space has failed

Had a really weird problem with one of our clients today. They have a Win 2012 Server (Hyper-V host) with 4 VMs. One VM is the DC, the other 3 have a Line Of Business (LOB) apps with SQL server instances. All was working fine but some of the automatic background processes on one of the LOBs were unable to access the Internet. Internally client PCs were able to connect to the LOBs MS-SQL DB without any issues.

When we checked this server via the Hyper-V console session, we noted that the DNS IPs in the network adapter had gone. I re-added the DCs (its also the DNS) IP, but this did not resolve the issue. I was unable to access the server over the network (RDP) or access other PCs from within the server's console. Yet the users were happily using the LOB and associated database!

A quick reboot would probably resolve the issue, but this was out of questions as users were seeing clients and needed their LOB application. Eventually I found this warning in the event log:

After using a bit of Google foo, we came up with some answers, thanks to all the posters! This gave us the most background info:
http://blogs.technet.com/b/askds/archive/2008/10/29/port-exhaustion-and-you-or-why-the-netstat-tool-is-your-friend.aspx
Clearly, one of the processes running on this server is not a good citizien! Running "netstat -anob" produced a 3.5MB file! This blog entry:
http://kasperk.it/windows-server/a-request-to-allocate-an-ephemeral-port-number-from-the-global-udp-port-space-has-failed-due-to-all-such-ports-being-in-use
suggested to kill the offending process, but the svchost.exe was also running lanman Workstation, which I thought was a big risk, gen the system was still operational for the end-user.

We scheduled a suitable time to rectify the issue and I then killed the "dnscache" task. Low and behold the network sprung back to life and the system performed normally. The LOB application which I left running on 1 PC noted a sub-second interruption but recovered transparently. For good measure we restarted the system.

What is the cause of the port exhaustion? At this stage I don't know, but I will keep an eye out. The server had been running without an issue for 5 months. It was last restarted 6 days ago when I installed Windows Updates (June 2013). There is likely an issue with the update and one of the programs on the server.

Quickbooks / Reckon Accounts Firewall rules

In a previous blogpost, I discussed setting the correct firewall port rules for Quickbooks. In Australia the developer of Quickbooks has parted ways from the American mothership and the product has been re-branded Reckon Accounts. Who knows what is going to happen with the program over time.

Rather than port numbers I now use the executable names for the firewall exceptions. When it comes to annual upgrades, you just have to change the file/folder names, rather than guessing port numbers or call their support desk.

I have developed a script that sets the right options:

REM Add firewall rules for Reckon Accounts 2013
REM Explanations can be found at
REM http://technet.microsoft.com/en-us/library/dd734783(v=ws.10).aspx

netsh advfirewall firewall add rule name="Reckon Accounts DB Service" profile=Domain,Private dir=in action=allow program="C:\Program Files (x86)\Intuit\ReckonAccounts 2013\QBLanService.exe"
netsh advfirewall firewall add rule name="Reckon Accounts DB Manager" profile=Domain,Private dir=in action=allow program="C:\Program Files (x86)\Intuit\ReckonAccounts 2013\QBDBMgrN.exe"
netsh advfirewall firewall add rule name="Reckon Accounts DB Manager (N)" ;profile=Domain,Private dir=in action=allow program="C:\Program Files (x86)\Intuit\ReckonAccounts 2013\QBDBMgrN.exe"
pause

Cut and paste the above into notepad and save as a command file (.cmd), then run it and the firewall rules for the server are set!

Adding an alias in Office 365 (Wave 15, P & M plans)

I recently needed to assign an email alias for a user. In Wave 14 administrators were able to set "extra" Exchange options for an Office 365 users mailbox through the standard web interface:

However in Wave 15 I was unable to find these settings for customers on P or M plans (right hand picture, whereas E plans (left) still have the option:

 

After a bit of Google searching I found a neat trick, how you get the Wave 15 interface to display the "Exchange Control Panel" (ECP). Simply logon to Office 365 as the user who's mailbox needs further options set. Then change the URL from
https://pod12345.outlook.com/blah/blah?blah&blah....
to
https://pod12345.outlook.com/ecp

Voila! Now you have access to the extra Exchange options!

 

---

Update - July 2013
Due to the dynamic nature of Office 365, things change... The above option and trick is no longer available for end users. However, administrators can now once again add additional email addresses for users. Simply select/edit the user account, then use the edit 'email addresses' link!!

Mysterious cab files fill-up temp folder

Our Labtech RMM recently spit out a warning that a client's 2008R2 server had their disk space filling up fast (Note they are not a premium customer):

Upon closer inspection I found that every hour an unknow process would attempt to write a .cab file of approx 60MB to the Windows temp folder. Checking with Process Explorer I found that it was makecab.exe writing these files. Makecab was invoked by services.exe, so that was a bit of a dead end. I looked through the list of Windows scheduled tasks, but did not find anything that was supposedly run every hour.

I tried to rename the cab files created, adding a .cab extension, but they turned out corrupt. I then used Sysinternals ProcessExplorer to find the source of the cab file, which was tricky, as it would only take a few minutes to actually create the file. The indication was that the source of the cab file (eg the file being compressed) was from the c:\windows\logs\cbs folder.

I learned a bit more about Windows Resource Protection (WRP), which stops programs overwriting essential system files. It keeps its log files in this folder. The SFC.exe program writes the details of each verification operation and of each repair operation to the CBS.log file. The CBS.persist.log is generated when the CBS gets to be around 50Mb in size. CBS.log is copied to cbs.persist.log and a new cbs.log file is started. A bit of Google foo and we determine that the cbs logs would only be useful for serious troubleshooting issues. If the system is running fine, we can delete this file. SFC.exe will create a new one, next time it is run.
So why do we have this mysterious process writing a file here?? It appears that on this server the SFC archive process has not been running for a long tome. On the 9th Feb the system was restarted and the log got recycled. Due to the long period of not being archived the log file grew to 4GB in size. Now the fun begins: Every hour, the archive process tries to create a new .cab file. I now speculate that the file size is larger than  what is supported and the process fails, hence resulting in a partial .cab file that sits in the temp folder, rather than a complete .cab file in the CBS log folder.
I have deleted the offending .cab file and most of the other ones too, just keeping a few recent ones in case we need them. No more mysteries!

Migrating to Exchange 2010

I recently performed an Exchange 2010 migration from a different domain running Exchange 2007. The whole migration worked much smoother than I expected... Here is an account of what I did.

Some user's at the client had large Exchange mailboxes with > 4 GB data. I installed a 30 day trial copy of Backup Assist (BA), my favourite SMB backup solution on the source server. Part of the BA suite is an Exchange mailbox backup. I first made sure that the program is working okay, as there are always a few hurdles to overcome with setting up the right Exchange permissions for the user account that performs the backup.

On the days before the migration I performed a "manual" mailbox backup with the destination grouping method set to "yearly backup". This option exports all the user's mailbox contents to a number of .pst files, separated into folders for each calendar year.

I then created a batch file that renamed these files, appending the year and moving them to the users's home share on the new server. The most current year (2011) I kept for migration to the new Exchange server. On the night of the server migration, I ran the standard BA mailbox backup that archived all emails to the 2012-Q1 folder and transferred this folder to the new server.

I now had to import the user's last two (2011 and 2012) .pst files into the new Exchange 2010 server. This server was installed with SP1 and patched with the latest roll-up. All user's were then setup and email flow tested. After the change-over emails were already being received by the new server. Exchange offers the new "New-MailboxImportRequest" command that make merging the contents of a .pst file into a mailbox a smooth process.

When I used the command from a PS command prompt, I encountered an issue with the permission:

The term ‘New-MailboxImportRequest’ is not recognized as the name of a cmdlet, function, script file

It turns out that by default no account has the right to run this PS cmdlet. To enable it you need to assign a new role to a specific group. I added the rights for the domain admins by using this command:

New-ManagementRoleAssignment -Name “Import Export PST” -SecurityGroup “Domain Administrators” -Role “Mailbox Import Export”

Now we were ready to begin the actual import/merge of the user's .pst files:

New-MailboxImportRequest -Mailbox -FilePath \\server\d$\Users\Mailbackup\2011\user.pst

This command was followed by a second set for all the 2012 email .pst files. I created another "batch" file, except this one was full of the above Powershell commands. Once I started this Powershell script the merge process took a good 1-2 hours. Along the way you can check progress by accessing the import queue using

Get-MailboxImportRequest

or to get more details

Get-MailboxImportRequest -Status Failed | Get-MailboxImportRequestStatistics -IncludeReport | Format-List > c:\netadmin\pstimport.txt

I encountered a few errors that were detailed by the above command. They all related to an issue with the mailbox message size:

Message (size 15.71 MB (16,474,863 bytes)) exceeds the maximum allowed size for
submission to the target mailbox. You can increase this limit by using the Set-Mailbox cmdlet in the Exchange Management Shell.
Error details: MapiExceptionMaxSubmissionExceeded: Unable to save changes. (hr=0x80004005, ec=1242)

I fixed this on the affected user's account by going to the user's mailbox properties:

1. Go to mail flow settings, then set Message Size Restriction properties and set both Sending and Receive message size to a high value, say 50Mb.
   
2. Re-run the failed "MailboxImportRequest" again, it should now be successful.
3. Reverse the user's message size restriction back to its default value.

Sure there would have been a PS script option to perform his, but for those few users affected, the Set-Mailbox options looked to overwhelming!

Overall I have found this approach to be much easier than trying to move mailbox databases from one domain to another. The other advantage is that user's mailboxes are archived and pruned. If the user needs an old .pst file, they can attached it themselves to their Outlook.

Reference: MS Exchange .org article

I have been a member on Experts Exchange for some time. Today they announced a major facelift of their site. Check it out by clicking here: