Mysterious cab files fill-up temp folder

Our Labtech RMM recently spit out a warning that a client's 2008R2 server had their disk space filling up fast (Note they are not a premium customer):

Upon closer inspection I found that every hour an unknow process would attempt to write a .cab file of approx 60MB to the Windows temp folder. Checking with Process Explorer I found that it was makecab.exe writing these files. Makecab was invoked by services.exe, so that was a bit of a dead end. I looked through the list of Windows scheduled tasks, but did not find anything that was supposedly run every hour.

I tried to rename the cab files created, adding a .cab extension, but they turned out corrupt. I then used Sysinternals ProcessExplorer to find the source of the cab file, which was tricky, as it would only take a few minutes to actually create the file. The indication was that the source of the cab file (eg the file being compressed) was from the c:\windows\logs\cbs folder.

I learned a bit more about Windows Resource Protection (WRP), which stops programs overwriting essential system files. It keeps its log files in this folder. The SFC.exe program writes the details of each verification operation and of each repair operation to the CBS.log file. The CBS.persist.log is generated when the CBS gets to be around 50Mb in size. CBS.log is copied to cbs.persist.log and a new cbs.log file is started. A bit of Google foo and we determine that the cbs logs would only be useful for serious troubleshooting issues. If the system is running fine, we can delete this file. SFC.exe will create a new one, next time it is run.
So why do we have this mysterious process writing a file here?? It appears that on this server the SFC archive process has not been running for a long tome. On the 9th Feb the system was restarted and the log got recycled. Due to the long period of not being archived the log file grew to 4GB in size. Now the fun begins: Every hour, the archive process tries to create a new .cab file. I now speculate that the file size is larger than  what is supported and the process fails, hence resulting in a partial .cab file that sits in the temp folder, rather than a complete .cab file in the CBS log folder.
I have deleted the offending .cab file and most of the other ones too, just keeping a few recent ones in case we need them. No more mysteries!

Migrating to Exchange 2010

I recently performed an Exchange 2010 migration from a different domain running Exchange 2007. The whole migration worked much smoother than I expected... Here is an account of what I did.

Some user's at the client had large Exchange mailboxes with > 4 GB data. I installed a 30 day trial copy of Backup Assist (BA), my favourite SMB backup solution on the source server. Part of the BA suite is an Exchange mailbox backup. I first made sure that the program is working okay, as there are always a few hurdles to overcome with setting up the right Exchange permissions for the user account that performs the backup.

On the days before the migration I performed a "manual" mailbox backup with the destination grouping method set to "yearly backup". This option exports all the user's mailbox contents to a number of .pst files, separated into folders for each calendar year.

I then created a batch file that renamed these files, appending the year and moving them to the users's home share on the new server. The most current year (2011) I kept for migration to the new Exchange server. On the night of the server migration, I ran the standard BA mailbox backup that archived all emails to the 2012-Q1 folder and transferred this folder to the new server.

I now had to import the user's last two (2011 and 2012) .pst files into the new Exchange 2010 server. This server was installed with SP1 and patched with the latest roll-up. All user's were then setup and email flow tested. After the change-over emails were already being received by the new server. Exchange offers the new "New-MailboxImportRequest" command that make merging the contents of a .pst file into a mailbox a smooth process.

When I used the command from a PS command prompt, I encountered an issue with the permission:

The term ‘New-MailboxImportRequest’ is not recognized as the name of a cmdlet, function, script file

It turns out that by default no account has the right to run this PS cmdlet. To enable it you need to assign a new role to a specific group. I added the rights for the domain admins by using this command:

New-ManagementRoleAssignment -Name “Import Export PST” -SecurityGroup “Domain Administrators” -Role “Mailbox Import Export”

Now we were ready to begin the actual import/merge of the user's .pst files:

New-MailboxImportRequest -Mailbox -FilePath \\server\d$\Users\Mailbackup\2011\user.pst

This command was followed by a second set for all the 2012 email .pst files. I created another "batch" file, except this one was full of the above Powershell commands. Once I started this Powershell script the merge process took a good 1-2 hours. Along the way you can check progress by accessing the import queue using

Get-MailboxImportRequest

or to get more details

Get-MailboxImportRequest -Status Failed | Get-MailboxImportRequestStatistics -IncludeReport | Format-List > c:\netadmin\pstimport.txt

I encountered a few errors that were detailed by the above command. They all related to an issue with the mailbox message size:

Message (size 15.71 MB (16,474,863 bytes)) exceeds the maximum allowed size for
submission to the target mailbox. You can increase this limit by using the Set-Mailbox cmdlet in the Exchange Management Shell.
Error details: MapiExceptionMaxSubmissionExceeded: Unable to save changes. (hr=0x80004005, ec=1242)

I fixed this on the affected user's account by going to the user's mailbox properties:

1. Go to mail flow settings, then set Message Size Restriction properties and set both Sending and Receive message size to a high value, say 50Mb.
   
2. Re-run the failed "MailboxImportRequest" again, it should now be successful.
3. Reverse the user's message size restriction back to its default value.

Sure there would have been a PS script option to perform his, but for those few users affected, the Set-Mailbox options looked to overwhelming!

Overall I have found this approach to be much easier than trying to move mailbox databases from one domain to another. The other advantage is that user's mailboxes are archived and pruned. If the user needs an old .pst file, they can attached it themselves to their Outlook.

Reference: MS Exchange .org article

I have been a member on Experts Exchange for some time. Today they announced a major facelift of their site. Check it out by clicking here:

Exchange 2010 Mailbox Recovery

Ooh what a pain.... when you don't know how to do this. Today I learned the hard way!

Today I had to use Exchange recovery on an SBS 2011 server. It started with the client requesting that the user account be renamed from username "Fred" to a role based name "Pleb". For some starnge reason the user was no longer able to log on after I had renamed the account, so I simply deleted "Pleb" and recreated the accout from scratch. The user then demanded his email back and I thought, easy, just reconnected form the "Disconnected Mailbox" folder in the EMC. Hmm - long face, there was no such folder. It appears that if you create a new account with the same name that an existing account had, the disconnected mailbox is not reccyled or being left - no SBS deletes it. Have to try this on a real Exchange server to check the outcome. Anyway, I was facing a rstore from backup - my first foray into Exchange 2010 recovery!

This SBS server is a VM, and backups are done on the physical server using Backup Assist. When I recovered the the Hyper-V machine, I found the partition with the mailbox data file, but this is slightly different from an actual Windows application image backup on a Exchange server.

I restored the mailbox.edb file to a temp folder, but I had no log files. I fired up the exchange Management Shell (EMS) and ran

Eseutil /mh "E:\Temp\db\Mailbox Database.edb"

This command gives you some stats on the DB, the important one being the "State". in my case it was 'Dirty Shutdown', expected but not that desirable. Note to self, need to check with backup Assist support how I can get the log files. Without log files I had to perform a hard repair using:

Eseutil /p "E:\Temp\db\Mailbox Database.edb"

If we have had the log files, we could simple get the "E

Eseutil /r "E03" /l e:\temp\log /d e:\temp\db /s e:\temp\rec

I then repeated the

Eseutil /mh "E:\Temp\db\Mailbox Database.edb"

command, which brought the database back to a state of 'Clean'. The next step now was to bring this recovery database online, using:

New-MailboxDatabase RecDB -Recovery:$true -EdbFilePath "e:\temp\db\Mailbox Database.edb" -LogFolderPath "e:\temp\log" -Server Alpha

This will show the database in the EMC, but it's state is still 'unmounted'. Therefore we issue the command

Mount-Database RecDB

which mounts the database, followed by the command

Get-MailboxStatistics -Database RecDB

which lists all mailboxes in my recovery DB. My next step took some time to research as I had two options to consider: a) export the mailbox to a PST fle or b) recover the mailbox back into a users mailbox. Option a) looked quite attractive as I would not have to fiddle with the new mailbox, but it turns out that there are quite a few pre-requisites for doing this. I therefore opted for the second solution, using a command to recover the contents of the backup/recovery database into a special folder "restore" of the user's mailbox:

Restore-Mailbox -Identity "Fred" -RecoveryDatabase RecDB -RecoveryMailbox "Pleb" -TargetFolder Restore

This caused a prompt to appear, asking for confirmation to import those items into the existing mailbox. Logging on as the user, I did indeed find the new "Restore" subfolder with the desired mail items inside. Phew! What remains to do do is a bit of clean up and house keeping:

Dismount-Database RecDB

then follwoed by

Remove-Mailboxdatabase RecDB

and finally

Delete "E:\Temp\db\Mailbox Database.edb"

Happy restores!

SBS Monitoring with high CPU load

Just encountered an issue with an SBS 2008 server. High CPU load was caused by the SBSMonitoring SQL instance. Eventually narrow it down and applied the fix described here:

http://social.technet.microsoft.com/Forums/en/smallbusinessserver/thread/d195baac-da8b-4387-9079-c55d5e1879b4

Whilst checking the event log afterwards, I also noted these information events. Another Google search promised resolution by using this fix:

http://social.msdn.microsoft.com/Forums/en/sqldatabaseengine/thread/0b3a2163-929f-4ba6-8e74-3d683aea19f1

Remote Installation Service

Today I finally resolved a problem that bugged me for some time. A 2003 Server has been used to install Win XP to PCs on the network using the Windows Remote Installation Service (RIS). The RIS service was upgraded by Microsoft with the release of 2003 Service Pack 2 and is now called Windows Deployment Service (WDS). I recently started working through Win7 deployment scenarios. In the process I changed WDS from "legacy" mode to "mixed" mode.

When I tried to install WinXP again via RIS, I received an error:
PXE-T01 error > file not found TFTP

This message indicates that the client PC received at least one valid DHCP/BOOTP offer, but does not have a boot filename to download. After much head-scratching and Googling, I finally compiled the info to make it all work:

Add these DHCP Options:
060 PXE Client - enter: "PXECLIENT"
066 Boot server host name - enter the IP of the RIS/WDS server
067 Bootfile name - point to: "remoteinstall\oschooser\i386\startrom.com"

By default, DHCP option 60 is not shown on the server and needs to be added to the scope. If WDS and DHCP run on the same server, this can be done through the properties of the WDS, but since my DHCP was not I had to add it manually from a Command Prompt:

1. C:\Windows\System32>netsh
2. netsh>dhcp
3. netsh dhcp>server \\server_machine_name
4. netsh dhcp>add optiondef 60 PXEClient String 0 comment=PXE support
5. netsh dhcp>set optionvalue 60 STRING PXEClient
6. netsh dhcp>exit

Then return to the WDS admin tool and expand it to get to the Properties dialog for the server. Click the 'Boot' tab and enter 'OSChooser\i386\startrom.com' under the x86 architecture.

Restart both the WDS and the DHCP service and you should be able to boot from the network!

Turn off Server Licensing Trace Log files

In an attempt to save disk space on the C: drive of an SBS 2008 server I have found this article on how to turn off trace logs for licensing. They do not occupy a huge amount of space (it was 400MB on the server I noticed), but sometimes every bit counts...

http://technet.microsoft.com/en-us/library/cc940957%28WS.10%29.aspx

The article refers to the EBS product, so only the references to ServerInfrastructureLicensing are applicable to SBS 2008