The issue has been reported by a handful of users on the Adobe Forums and some of my users ran into it after I began a test roll-out of Acrobat on our site yesterday.

These are the exact error messages we’re seeing:

Microsoft Visual C++ Debug Library 

Runtime Error! 

Program: C:\Program Files\Adobe\Acrobat 9.0\Acrobat\Acrobat.exe 

The application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.

and when attempting to view PDF files in Internet Explorer:

Microsoft Visual C++ Debug Library 

Debug Error! 

Program: C:\Program Files\Internet Explorer\iexplore.exe 

abnormal program termination 

(Press Retry to debug the application)

We often see odd problems with software running on our network due to the various configuration options we use that set our workstations apart from the average home computer (and a lot of computers in other schools, for that matter), and often trip up programs that haven’t been tested on domain workstations. I ran a trace using ProcMon to see if I could work out what was going on:

98959 11:29:15.6155113 Acrobat.exe 972 CreateFile \Device\Mup\ NAME INVALID Desired Access: Read Data/List Directory, Synchronize, Disposition: Open, Options: Directory, Synchronous IO Non-Alert, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a
99021 11:29:15.6181294 Acrobat.exe 972 CreateFile \\mydomain\users\ SUCCESS Desired Access: Read Data/List Directory, Synchronize, Disposition: Open, Options: Directory, Synchronous IO Non-Alert, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a, OpenResult: Opened
99100 11:29:15.6219431 Acrobat.exe 972 QueryStandardInformationFile \\mydomain\users\ SUCCESS AllocationSize: 0, EndOfFile: 0, NumberOfLinks: 1, DeletePending: False, Directory: True
99111 11:29:15.6222464 Acrobat.exe 972 CloseFile \\mydomain\users\ SUCCESS
99182 11:29:15.6254897 Acrobat.exe 972 CreateFile \\mydomain\users\ NAME COLLISION Desired Access: Read Data/List Directory, Synchronize, Disposition: Create, Options: Directory, Synchronous IO Non-Alert, Open Reparse Point, Attributes: N, ShareMode: Read, Write, AllocationSize: 0
99187 11:29:15.6257927 Acrobat.exe 972 RegOpenKey HKU\S-1-5-21-1708537768-2052111302-682003330-4640\Software\Adobe\Adobe Acrobat\9.0 SUCCESS Desired Access: Read
99189 11:29:15.6258324 Acrobat.exe 972 RegQueryValue HKU\S-1-5-21-1708537768-2052111302-682003330-4640\Software\Adobe\Adobe Acrobat\9.0\Crashlog NAME NOT FOUND Length: 144
99191 11:29:15.6258536 Acrobat.exe 972 RegCloseKey HKU\S-1-5-21-1708537768-2052111302-682003330-4640\Software\Adobe\Adobe Acrobat\9.0 SUCCESS
99192 11:29:15.6259012 Acrobat.exe 972 QueryNameInformationFile C:\Program Files (x86)\Adobe\Acrobat 9.0\Acrobat\Acrobat.exe SUCCESS Name: \Program Files (x86)\Adobe\Acrobat 9.0\Acrobat\Acrobat.exe
99193 11:29:15.6260191 Acrobat.exe 972 RegOpenKey HKLM\Software\Wow6432Node\Microsoft\Windows\Windows Error Reporting\Escalation NAME NOT FOUND Desired Access: Read

The suspicious line here is in red: Acrobat issued a CreateFile on the root of the DFS share hosting the redirected AppData, with CreationDisposition set to CREATE_ALWAYS; in other words, trying to overwrite the share… unsurprisingly this resulted in a NAME COLLISION since it already exists, and a call to Windows Error Reporting immediately followed.

As I had tested the software on my own laptop I first assumed that perhaps only non-Administrators were affected, but quickly put paid to that theory after checking in with a non-admin who was also testing the software who only had the problem on certain machines. After checking in with further users, I found that only our desktop workstations were affected, not laptops, regardless of which user logged in. Since the same user was affected on a desktop but not a laptop, I surmised the difference was likely to do with something specific the fact that our desktop computers are configured to redirect most users’ Application Data directory to a network share, in order to keep consistent program settings regardless of where they log on. The laptops are not configured in this way, so that users can more easily access their accounts while off-site and not connected to the network.

Taking a page from Mark Russinovich’s book (or blog, to be more precise), I setup one of my Windows Vista testbench workstations to generate a crash dump for Acrobat in order to analyse, and from reading through the stack trace it appeared the problem stems from the PDFLTerm() function in AdobePDFL.dll (remember this is a stack, so the cause of the issue is usually below all the exceptions):

ntdll!KiFastSystemCallRet
ntdll!ZwWaitForMultipleObjects+0xc
kernel32!WaitForMultipleObjectsEx+0x11d
kernel32!WaitForMultipleObjects+0x18
kernel32!WerpReportFaultInternal+0x16d
kernel32!WerpReportFault+0x70
kernel32!UnhandledExceptionFilter+0x1b5
msvcr80!abort+0xeb msvcr80!terminate+0x4d
Acrobat+0x422e
ntdll!__RtlUserThreadStart+0x6f
ntdll!_EH4_CallFilterFunc+0x12
ntdll!_except_handler4+0x8e
ntdll!ExecuteHandler2+0x26
ntdll!ExecuteHandler+0x24
ntdll!KiUserExceptionDispatcher+0xf
kernel32!RaiseException+0x58
msvcr80!_CxxThrowException+0x46
Acrobat_69630000!DllCanUnloadNow+0x68e27
Acrobat_69630000!PDFLTerm+0x645ce
Acrobat_69630000!DllCanUnloadNow+0x663df msvcr80!_NLG_Return

Armed with all of this, I quickly located the forum post I mentioned earlier where other users had found the same issue with Adobe Reader 9.0 under the same circumstances. As users of the free download of Reader, they were out on their own as Adobe have no obligation to give them one-to-one support. However, as someone who had just spent several thousand pounds buying Adobe software, including Acrobat, they would have to talk to me, so I called their UK support line.

As soon as I read out the error message to the agent I talked to (the very helpful Andrew Mitchell, for reference), he told me that he’d heard the exact same problem yesterday and that Adobe were aware of a problem producing this error. I mentioned my diagnostics and told him I suspected the AdobePDFL.dll, which he confirmed was the case. He put me on hold for a couple of minutes to talk to a senior technician, and when he returned, confirmed more of what I’d already found out.

Unfortunately, Adobe were only made aware of the issue very recently, and there is currently no ETA for a fix. I later got a follow-up email from Andrew which stated the following:

“Unfortunately because of the new-ness of the software we are unable to provide you with a direct solution to this issue as all solutions are experimental at this time.”

Translation: we’re still testing the patch for this. I was told I will be contacted again when a fix is available, but for the meantime, I would strongly advise you do not deploy Adobe Reader 9.0 or Adobe Acrobat 9.0 if your users’ have their Application Data directory stored on a network share.

That said, I am now going to rant about my #1 pet peeve about teachers.

It’s fair to say that teachers’ holidays are somewhat infamous. The general public views them as having an enormous amount of holiday time, in particular the 5-6 weeks during the summer, but also the week or two at Easter, Christmas, and the week in the middle of each term. On top of that, they have their mystical short hours because ‘they go home at 3.45′. Most people who don’t work in education do not factor in the late hours many teachers put in at home after school on a regular basis, or that many do lesson preparation during the aforementioned holidays. Teachers themselves are the first to rant whenever there’s a discussion about ‘how easy teachers have it with all that holiday’, which is why what I am about to describe pisses me off even more.

Whenever school restarts after a break (and I guarantee this will happen on Monday, the first day back after half term) someone will ask me, “Did you have a nice break?”

No, I did not have a nice break. I did not have a break at all. I know you were just making conversation, but as IT support we are, along with the caretakers, cleaners, the Business Manager, and the Bursar, on full time contracts. We are here through Easter, Christmas, half-terms, and the entire Summer Holiday. Also, do you remember what it was like the last two terms when the heating was broken? Well, during the breaks, the entire school’s heating gets turned off entirely, so in the winter breaks, any work we have to do outside our offices is done in coats and hats.

What’s more, last Monday morning, a teacher did actually come in to do some preparation, as they sometimes do. The first thing she asked me? “Are you actually doing any work, or just Facebooking everyone?” I was hungry and on my way to lunch so I gave one of my usual dry responses on the subject and carried on, but what I felt like saying was, “Actually I’m run ragged, but it’s obvious what your plans for the afternoon are.”

It’s this attitude that annoys me. Not only are we here, but the ‘holidays’ are when we are at our very busiest, because it’s during these short windows of time that we work our way through the usually gargantuan list of improvements to the IT provision that can’t be done during term because they are too disruptive to lessons, and because during term we spend most of our time running around fire-fighting because there are 1200 students and about 150 staff who at times seem hell-bent on breaking as much as possible. This stuff takes time, almost always more than we really have, so there’s an awful lot to do before the site is once again filled with people who take us for granted a lot more often than I’m comfortable with, need everything fixing right now, and are convinced that ‘it was fine before you did x so it must be your fault it’s broken’, a handful of typical attitudes which are little more than minor irritations when faced with the view that we sit around all day during the break sipping coffee and wasting time on Facebook.

I love my teacher friends, I really do. But it is only a matter of time before I kill one of them for making the same assumption about my ‘holidays’ that the rest of the country makes about theirs.

Last week I put in an application for a new position at work, and following an interview earlier this week, I have been appointed IT Support Manager at Challoner’s, effective 1^st February (i.e., next week).

To cut a long story short, I approached my boss a couple of weeks before Christmas to express my interest in doing further internal training and taking on more responsibility, although there was no real position to fulfil that as the IT roles that existed at Challoner’s were ‘IT Technician’ (the job I already had) and ‘Network Manager’ (the position which my boss isn’t planning to vacate any time soon). It turns out that senior management had seen this day coming and had already discussed what to do about it, so in quite short order the wheels started to turn to create a new position that could I slot into. Huzzah! etc.

High Contrast Mode is an accessibility feature in Windows that causes the desktop theme to switch off of Aero (or whatever is currently set) onto an almost completely black scheme with white text. It also disables the desktop wallpaper and changes the background colours of applications like Word to black, so it appears the user is typing white text onto black paper.

Despite repeated use of the Shift+Alt+PrtScrn shortcut, High Contrast Mode refused to turn off. Their desktop wallpaper did return, indicating that it was trying to switch out of High Contrast Mode, but not fully succeeding. Our students tend to have much of the Control Panel locked down (as I outlined above, the little bastards cherubs like to fiddle and break things), so the Control Panel applet to turn this off was not available to them, but even making it available and using the applet did not work.

I couldn’t find any mention of this via Google, and so it took me a little while to work out what was going on. I eventually traced this problem to an incorrect setting in the Registry. Here’s what my Vista laptop has set when High Contrast is activated:

[HKEY_CURRENT_USER\Control Panel\Accessibility\HighContrast]
"Flags"="126"
"High Contrast Scheme"=""
"Previous High Contrast Scheme MUI Value"=""
"Pre-High Contrast Scheme"="C:\Windows\resources\themes\Aero\Aero.msstyles"
"Pre-High Contrast Color"="NormalColor"
"Pre-High Contrast Size"="NormalSize"
"Pre-High Contrast Wallpaper"="C:\Users\JSCHLA~1\AppData\Local\Temp\BGInfo.bmp"

As you can see, this key records what the desktop scheme was set to before I switched to High Contrast Mode, in the “Pre-High Contrast Scheme” value.

Now let’s look at the registry of a user with the problem:

[HKEY_USERS\S-1-5-21-1708537768-2052111302-682003330-9422\Control Panel\Accessibility\HighContrast]
"Flags"="127"
"High Contrast Scheme"="High Contrast Black"
"Previous High Contrast Scheme MUI Value"="High Contrast Black"
"Previous High Contrast Scheme MUI Ptr"="@themeui.dll,-852"
"Pre-High Contrast Scheme"="@themeui.dll,-852"
"Pre-High Contrast Color"=""
"Pre-High Contrast Size"=""
"Pre-High Contrast Wallpaper"="C:\Users\BEN~1.KOL\AppData\Local\Temp\BGInfo.bmp"

As you can see there are a number of differences, but those of you paying attention will have immediately noticed the most pertinent discrepancy; this user’s “Pre-High Contrast Scheme”. Instead of the path to the Aero style file, it lists the previous scheme as “@themeui.dll,-852″. This is a direction to Windows to extract a string resource from a dll file and use that value as the theme. On an English Vista system, “@themeui.dll,-852″ contains the text string “High Contrast Black”. This means that when the user tries to deactivate High Contrast Mode, the scheme it reverts to is… the one used by High Contrast Mode.

How this happened is, so far, a mystery to me.

I corrected it using the following steps:

1. Ensure user is logged on and in ‘true’ High Contrast mode, by checking if their wallpaper appears. If there is no wallpaper, they are in High Contrast mode. If they are not, use the keyboard shortcut (Shift+Alt+PrtScrn) to switch it on.

2. Edit the user’s registry to set:

   HKEY_CURRENT_USER\Control Panel\Accessibility\HighContrast
   "Pre-High Contrast Scheme"="C:\Windows\resources\themes\Aero\Aero.msstyles"

   If RegEdit is prohibited for the user, either use Remote Registry, or create a shortcut to RegEdit on the desktop and use ‘Run as Administrator’, then set the same value under HKEY_USERS\SID-Of-User\Control Panel\Accessibility\HighContrast, substituting SID-Of-User for the SID of the user with the problem.

3. Use the keyboard shortcut again to switch out of High Contrast Mode. Theme should switch back to Aero.

4. Dance the happy dance of success (Optional).

5. Find child who turned High Contrast on as a joke and beat savagely with keyboard (Mandatory).

Hopefully that will help anyone with the same problem; I was getting surprising and unusual amount of stick from the kid’s teacher for “not fixing the problem” that was partly caused by a user, and partly (I presume) a bug in Vista. Such is life when you work IT Support; even if the user was completely responsible, it’s still your fault that it’s not fixed within minutes.

While tidying IT4 today, we found a long-lost IT technician in hibernation.

UPDATE 11.01: Staff and police have sealed off the rear of the site to pupils. The Head has just made a PA announcement that police are searching for someone on the run (presumably in the fields to the rear of the school), and that as a precaution, all students are being kept indoors and off the playgrounds and tennis courts during break (now).

UPDATE 11.32: Fire evacuation alarm sounded briefly, followed by an urgent announcement for everyone to remain inside. This was apparently caused by the builders, who are on site at the moment, digging through a fire alarm cable. This isn’t remotely surprising, as in the last week they have dug through a mains electricity cable, a telephone cable, and two different water mains.

We’re hearing reports of a heightened police presence all around the New Town area in Amersham.

UPDATE 12.02: Helicopter left about 15 minutes ago, but can still be heard in the distance along with sirens intermittently from the town. There’s a police unit stationed in the school car park still wearing an all-black short sleeved uniform. This isn’t the normal police uniform, and the last time I saw it in Reading train station, it was being worn by an armed officer. No sign of whether this is the case here or not.

UPDATE 12.18: We’ve just received an ‘all-clear’ from the police and the school timetable has been returned to normal.

After leaving at the appointed hour, there followed a trek down the M25 and M3. As we neared, we realised that we were passing an awful lot of signs for Birdworld, eventually culminating in driving right up to the entrance. Our driver Steve started to wind Mat up by putting his indicator on as we approached the turning, to which Mat exclaimed “It is Birdworld!”, only to fall back into his seat dejected when we drove straight past.

Instead, we ended up at a place called Go Ape, for what can be best described as a high-wire assault course. Of course, this had some implications which I felt were necessary to disclose to our team leader:

Me: “Janet, would this be a bad time to tell you that I’m scared of heights?”
Janet: “Yes.”

As it turns out, I’m not actually scared of heights, as I had though when getting very dizzy on top of an Army assault course scramble net several years ago. At the time I had attributed this to nearly dying aged 11 when I fell 6 feet out of a tree. However, it seems that I am in fact scared not of heights but of falling, so once strapped into the compulsory safety harness I was in fact one of the least worried people there. Mat got knackered and wimped out after the first 3 courses, but this was in fact an advantage to the rest of us as he spent to rest of the afternoon shooting photos, mostly of me looking cool:

Complete Photo Set by Mat

All said it was pretty good, especially considering we’d been expecting some kind of outdoor orienteering exercise in the rain (which luckily held off until just after we left). I later discovered there’s a centre very near Reading in Bracknell – I sense a possible birthday outing in the making.

I dare one of them to come and complain about missing files.

In other news, I gave blood today. It has since occurred to me that the proportion of blood taken is quite close to the proportion of the student storage drive that I liberated…

When I arrived on Monday morning I could tell it would be an interesting day before I even left the car park. The police were already parked up and some traffic cones were out reserving a spot for the fire brigade (I have no idea what the police were here for, and a fire engine seemed to be onsite at some point every day this week for an equally mysterious reason). Then within 10 minutes of getting to my office I discover that some poor bastard in the 6th form was killed in a car crash during half term.

5 minutes later, I notice that the support server has crashed.

It actually did the same thing last week, and JD thought he’d fixed it; although this was not the case I had a fair idea of how to kick it back into life. It’s also not the end of the world since no user-facing services rely on it.

I then made the mistake of checking my voicemail before going off to the main IT office to liaise with my co-worker Mat over our server issue. I had one message, informing me that at one of the other schools we support, absolutely no-one could log on. This was made worse by the fact that the school in question uses logon and file servers located on our site (we have a 100Mb fibre connection between our two sites) so it was almost certainly our problem in some way.

While I hunted for the correct virtual server host to restart our support server (since the documentation is all in JD’s head), Mat got some more information from the other school in dribs and drabs over the next 45 minutes. We’d already tested the core server they use and found it was running fine – I’d been heavily involved in the setup of their new server so I knew what to check. However, Mat eventually discovered that half their file system was now on a second server that I knew nothing about, evidently added in after my involvement had ceased. That server had crashed, with the exact same symptom as our support server. The two machines are not related in any way, so why these two and only these two were down remains a mystery to this day; neither have had any problems since their last reboot so further diagnosis will be waiting for us next week.

At 4.20pm we get a call informing us that some of the IP phones are not ringing when called from their external number. Callers hear ringing on their end, but the phones never ring in the office. This particular external number just happens to be the main parental contact number, so is fairly important. The biggest problem here is that nobody other than JD knows how the IP phone system works, no login details had been left for that system, and we don’t even know which server the system runs on.

This was all on Monday.

It’s fair to say we don’t normally have this many serious problems in a week, let alone one day. The server problems were seemingly resolved in less than 90 minutes and we worked around the phone problem early next morning, but it was a somewhat stressful first day for the boss to be out. I understand now why this is the first holiday JD has taken in the 18 months I’ve worked at Challoners. Luckily the rest of the week was filled with much less serious issues of the sort we deal with normally, and by Friday afternoon things were actually pretty smooth and quiet. There is, however, a somewhat vomit-inducing wiring bodge on our server room right now that I will almost certainly be ridiculed for next week, and for many months to come, stemming from the fact that JD hasn’t gotten around to labelling any of the voice patch panels yet…