techBLOGogy.net » admin

Exchange 2010 generating lots of transaction logs

admin — Thu, 06 Jan 2011 15:29:48 +0000

So this is an interesting one… Yesterday we noticed one of the Exchange 2010 servers we manage was running low on drive space on the transaction log volume. I initially was thinking the usual suspects would be the culprit; VSS had freaked out and the logs weren’t getting truncated appropriately. This, after all, happens sometimes. After a quick review of the backups, everything had completed successfully in recent history. Odd. After looking again at the transaction logs I then realized they were all from today! I then noticed that a new transaction log was being created every couple of seconds! Not cool! Exchange 2010 can be transaction log heavy, but not to the tune of a log per second; that’s just crazy. I checked to make sure the server wasn’t an open relay, and it wasn’t. I checked to make sure the firewall was locked down to only allow inbound email from their SPAM filter provider; it was. I checked the queue…. empty. Literally no email coming or going from the Exchange server (small shop of like 20 users, so this isn’t all that unusual at a given time). I then watched the Exchange database and noticed that it wasn’t growing at all. So, something was asking the Exchange server to do something, which was generating a ton of logs… but wasn’t sending / receiving email and there wasn’t an influx of mail being dumped into a users mailbox from a PST import or something to that affect.

Here’s where it gets fun. I logged into the server after hours and rebooted the server. I was, at this point, assuming that it was a hung Exchange process or something so I didn’t bother to check if it was still exhibiting the behavior when I first logged in. After the reboot however, the Exchange server started acting normally again. One or two transaction logs per minute or so. Chalk it up to something being freaked out. Out of curiosity, I logged back in this morning and it was doing it again… and after a brief discussion with one of my coworkers who had checked it earlier this morning as well, we determined that it must have started again around 8:30. HAS to be a user related event.

I did some digging and found a few tools that can help us out in this situation; one stood out from the rest. Our old friend exmon. Turns out the Exchange Team has kept this tool up-to-date and it works just fine with Exchange 2010. Simply install the msi, browse to the install directory and run the .reg file to create the appropriate registry settings, and we’re off to the races. Here is what you get:

A listing of all your current Exchange users and exactly what they’re doing from a resource perspective. Note that my top talker is currently chewing on 68% of the CPU and asking the server to process about 19Mb of data… compared to the 716k of its next closest competitor, and around 4k from everyone else. Now, I will say that it isn’t necessarily unusual for a user to show up on the top of this list for one reporting period as using a nice chunk of CPU. This will happen if the user has just opened Outlook or has just performed a Send / Receive and there was a decent amount of email that needed to be downloaded. However, this user was sitting around 60-90% of the CPU for over 20 minutes. THAT is not normal.

So what was the culprit? This is a recent Exchange migration (we moved them from a hosted solution to their own Exchange server), and the PSTs had been imported. For whatever reason this users PST import had gotten stuck. The only sign that something was wrong was that the user’s Outlook constantly said “Folder is waiting to update”. Oddly enough, you could reboot the machine and as soon as the user logged in again and launched Outlook, the loop would start all over again. Deleting the user’s Outlook profile and recreating it solved the issue.

Fun times!

Another tool you’ll want in your toolbox is the Troubleshoot-DatabaseSpace.ps1 script provided with Exchange 2010 SP1. This can be an extremely useful tool in an emergency situation (i.e. you’re in very real danger of running out of disk space in the near future), however for my situation it would have been overkill as we had a couple days worth of disk space available to us easily. The big takeaway from the linked article is #4. If it finds an offending mailbox it will lock it out for 6 hours; meaning the user will no longer have access to their email. Again, since we had some time / disk space, and the customer hadn’t noticed any thing from a performance standpoint… this would have caused more disruption then we were currently experiencing.

–Dan Thompson

It’s Baby Time!!!

admin — Tue, 03 Feb 2009 22:24:49 +0000

Real quick before my mom passes out, my wife and I are not pregnant. It does, however, seem that a LOT of my friends are having babies… so this article is for them. So again, no announcements at this time.

As I mentioned though, it seems like a lot of my friends are having babies and subsequently posting ultrasound pictures (including the super cool 3D ones) to their websites and Facebook pages. In all their excitement however, the

soon-to-be parents are overlooking the fact that mommy’s social security number is on the image. Doh! Luckily the number (at least on the mishaps that I’ve seen) is some what inconspicuous in the fact that it doesn’t say “SSN” or “social security number”, so most viewers of the image will hopefully be so excited about the baby pictures that they’ll overlook it… and hopefully no “bad guys” are out on the internet trolling for ultrasound pictures (clearly its not the easiest way to steal someone’s identity, but crimes of opportunity happen all the time). That’s a chance I’d rather not take though, and I’m sure neither would the individuals involved; so here’s your friendly reminder… make sure and take a look at those pesky numbers all over the image prior to posting, and crop out any unnecessary personal info that doesn’t need to float around the internet for ages to come.

– Dan Thompson

More on the National Championship in 3D

admin — Mon, 05 Jan 2009 14:06:18 +0000

A few days ago I discussed the fact that the BCS National Championship game between the Oklahoma Sooners and the Florida Gators was going to be broadcast live in 3D here in Knoxville. I’ve since sat down with my friends over at WBIR and recorded a spot about the whole thing. Check it out!

– Dan Thompson

BCS Championship Game in 3D… in Knoxville!

admin — Mon, 29 Dec 2008 03:03:14 +0000

By now you may have heard that the BCS National Championship Game between the Sooners and the Gators will be broadcast in 3D to select theaters around the US on game day… if not… well the BCS National Championship Game is going to be broadcast in 3D to select theaters on game day! Oh, and we’ll be able to get in on the action here in Knoxville! Pretty cool stuff.

Sony Electronics and Fox Sports have teamed up with 3ality Digital, Cinedigm, and a slew of other technology provider big hitters to deliver the College National Championship game to 80 theaters in 35 different states,

one of which happens to be the Wynnsong Theatre in west Knoxville. 3ality, the same company that brought us U2 3D (yeah, I missed it too) earlier in the year, has been toying with broadcasting sports events (among other things) in 3D for some time now. On Dec 4 the company worked with its partners to broadcast the Raiders / Chargers NFL game to an invite-only showing at a handful of theaters in major US cities. This was such a hit that apparently the next step was the college National Championship game. The technology is interesting in that its signal rides over the same 2D infrastructure as other broadcasts instead of requiring a completely new one. This opens the possibility that you may one day be able to watch such events in your living room… providing of course that the TV manufactures step up and start creating 3D ready sets. In the mean time though, tickets for the event are $20 (get them here) and promises to be a fun time. As of this writing tickets are still available; let me know if you plan on attending!

– Dan Thompson

ASP.net and the Mailbox Server Role in Exchange 2007

admin — Tue, 23 Dec 2008 02:57:52 +0000

If you haven’t made the jump to Exchange 2007 yet, now is as good of a time as any. There have been tons of implementations already, a service pack, and several rollups since it was originally released to market… I think it’s safe to say that it’s a stable platform to move to. While you’re at it, go ahead and make the switch to Server 2008 as well. There is one sneaky little snag though, and for some reason mention of it seems to be missing from the Microsoft documentation as well as the other online HowTo’s I’ve thumbed through. If you’re like most cases I’ve been involved in, you’re not going to be installing Exchange 2007 in a completely new-to-Exchange environment. Chances are, this is going to be an upgrade / migration… and that’s where the missing link comes in to play. Have Entourage users? Yeah, this applies to you too.

Let me run this down for you. You’ve got several hundred mailboxes, of which a large majority of the users use webmail at some point in their email experience. Each of those users has conveniently bookmarked your company’s webmail address and it probably looks something like yourdomain.com/exchange… right? By now you’ve read that Exchange 2007 treats the “exchange” virtual directory as a legacy directory and is using “owa” instead for Outlook Web Access. Now we don’t want to have to go through and somehow convince all those users to update their bookmarks after you’ve migrated their mailbox over to the new servers (more importantly field the phone calls from users who didn’t listen the first 6 times you told them). You’ve done your homework and decided that the easiest migration path is to implement a client access server (CAS) separate from the mailbox server so you can take advantage of the auto redirecting (read: users with 2007 mailboxes can still browse to /exchange and CAS will automatically redirect their session to /owa… it’s seamless to them) among other things. Great idea… but it doesn’t work if you follow the directions given by Microsoft. Why? They don’t mention the need to add ASP.net to the web server role on the mailbox server.

So what happens without it? Users with a mailbox on the Exchange 2003 servers can login just fine. Users who have their mailboxes on the Exchange 2007 server and browse directly to the /owa address can login just fine. However, users who have a mailbox on the Exchange 2007 servers and browse /exchange get told their username and password is incorrect. Sound familiar? Oh, and wait a minute… lets not forget about our beloved MAC users. You migrated their mailboxes and now they don’t work either (this, after you spent hours trying to get the certificate installed correctly… don’t worry, I’ll discuss that in an upcoming article). Entourage fires up, appears to connect but never downloads any messages or displays the default (or any other) folders. Turns out that Entourage 2004 and 2008 are both hardcoded to send traffic to that /exchange virtual directory… no matter how you configure the client… and Exchange needs ASP.net to properly handle the DAV request. Neat huh?

Already have Exchange installed? All is not lost. Just simply fire up the Server Manager and add the role service to your web server role… or better yet, from the command prompt type ServerManagerCMD -i Web-ASP-net and hit enter (it’s okay guys… the black screen with white letters won’t hurt you). Reboot the server for good luck (in reality an iisreset will do the job), and you’re off to the races.

Happy Messaging!

– Dan Thompson

Last Minute Gift Ideas… for Techies!

admin — Fri, 19 Dec 2008 21:58:36 +0000

Around this time of year I can always count on being quizzed by friends and strangers alike on what to get the techies on their Christmas list. Admittedly this can be a daunting task, especially when you yourself are not technically inclined. Well, just for you guys, I sat down… racked my brain… and came up with a few “can’t go wrong” ideas. Check out my discussion with WBIR morning show’s Russell Biven on the topic, and let me know how these work out for you.

– Dan Thompson