Dan Griffin's Blog

Comments on security, PKI, smart cards, cryptography, and entrepreneurship.

My firm has been running Small Business Server 2003 for a few years now, and there’s no question that it’s been a life saver for us. SBS is a critical product for the industry.

But, in retrospect, given the nature of our business (Windows security development), we sort of outgrew SBS before we’d even deployed it. That is, our infrastructure demands have always been a little too sophisticated (e.g. too many client machines; need an enterprise CA; etc). It was the right decision at the time, though.

This post is really a continuation of a recent one in which I described the successful migration of a physical file server over to Hyper-V (HV).

One side note to that post: after that migration, the domain machine account of that server somehow got whacked, so I had to disjoin and rejoin that machine to the domain. Not a big deal, but I have no idea why that happened. Let me know if anyone else has seen that.

Anyway, that file server migration was really just a practice session for migrating the SBS machine. This is for the same big reason that I mentioned before: disaster recovery. The SBS is actually already protected by two separate backup solutions, one of which is offsite. But the machine itself is old, and system restoration is notoriously sensitive to hardware changes (i.e. in case I couldn’t find the same replacement hardware in a pinch).

Plus, the ultimate goal, once everything’s in HV, is to protect the VM images two ways: first, an onsite copy on a network share. This is for rapid recovery in a “minor” disaster scenario. Importantly, there’s no dependency on finding the same hardware. In theory, this should yield a full recovery in at most a few hours, probably much less.

The second backup plan is to mirror the VM images in the cloud, such as on Amazon’s S3 storage service. This addresses the major disaster scenario. While it might take a few days to copy the latest VM images down to a new location from the cloud (once the new location exists), that’s likely better than starting over, due to reasons discussed in my previous post.

One problem, though: for the life of me, I could not get the existing SBS image running in HV. I tried several different things, but I think it boils down to these two roadblocks:

First, the primary disk on the SBS is SCSI, and Dell (like many manufacturers) uses a small hidden EISA partition for configuration. The problem is that, contrary to what’s implied in the online forums regarding the behavior of VMware Converter, that tool doesn’t handle the EISA partitions correctly, in that the resulting primary partition isn’t bootable.

If you do decide to try that route, be sure to not attempt to convert the EISA partition and do fix-up the boot.ini in the resulting primary partition image. You’ll at least get as far as I did. But even after that I couldn’t get past a BSOD 0×7b in text mode boot.

The second roadblock is the affinity that SBS has for physical NICs and TCP/IP settings. There are lots of reasons for this, including the use of features such as Remote Access. To summarize, after giving up on VMware Converter, I tried setting up a clean SBS instance within HV and doing an NT restore of system files plus AD data.

If you happen to try that route, be prepared to reboot your system many times, since you’ll need directory restore mode for the AD restore, Safe Mode to get the virtual devices detected again, and possibly directory restore mode again in order to fix-up the network settings (see this post). The other hazard with this approach is that I found it difficult to troubleshoot the VM without it having access to the LAN, which is a problem since it implies the the original physical/production server must be off the LAN and its IP address temporarily migrated.

Ultimately, with the second approach, I couldn’t get the SBS VM to be fully responsive on the virtual NIC. I also noticed that the Windows networking subsystem still had hidden references to the old physical NICs, even though that hardware was no longer connected. More experienced SBS administrators may have work-arounds for either or both of those problems.

Also, for posterity, it’s possible that you could have greater success in only restoring AD data, but excluding system files. If you try that, you should probably run completly through SBS setup before doing the AD restore (rather than cancelling SBS setup and doing the full restore after the first logon, which is what the docs recommend). Potentially, that approach would give you all of the system files you need, while minimizing the risk of bringing in conflicting settings.

Anyway, I gave up on that approach. Instead, I introduced a separate (VM-based) Server 2008 DC into the existing SBS domain. These two posts (here and here) tell you pretty much everything you need to know.

This “extra DC” approach at least gives us some redundancy in case the SBS dies.

Permalink | Comments (1)

… but a recently published paper and demo drive the point home.

In summary, by exploiting collisions in the MD5 cryptographic hash algorithm, the authors were able to create bogus certificates signed by real/public/internet Certificate Authorities. As a result, signatures produced from those bogus certificates would automatically be trusted by a large number of client machines.

For example, I just performed a cursory examination of the root certificates currently trusted on my Vista SP1 machine, matching against the list in the above paper. I see that the “TC TrustCenter Class 3 CA” is trusted, and that it’s signature is indeed MD5-based. I also see one or more certs from Equifax and Thawte that meet the same criteria.

Why is this bad? It’s long been argued that trust decisions - such as what secure email code signing authorities to allow - should be taken out of users’ hands in an enterprise environment. That implies that those decisions are automated. For example, if software is signed by a certificate issued from such-and-such root CA, trust it and allow it to be installed.

But few enterprises have their own PKI, and even fewer can afford IT professionals who are qualified to administer it. Thus, the long list of root CAs that can be trusted by a typical client (Windows, Firefox, whatever) probably still are.

And home users are almost guaranteed to be vulnerable to this type of attack.

The assumption is that, if the cert chain is valid, then the intentions of the originator of that signature can be trusted. This in turn is partly based on the assumption that it’s not in the best interest of a typical public CA, especially the more well known ones, to issue certs to bad guys.

There are lots of reasons why those assumptions don’t work. This paper isn’t the best of those reasons, but it’s still an interesting one.

Permalink | Comments (0)

I recently successfully imported a physical line-of-business (LOB) server into Hyper-V. In other words, replacing a physical server with a virtual machine. That migration is a procedure that lots of businesses, small and large, will be encountering for reasons similar to my own.

What are those reasons? Well, they’re the usual rationale that favor virtualization. First, the LOB file server in question was getting old (three years). And while my firm has standardized on Dell server gear, that one was a custom job from a local shop. Eliminating old and/or non-standard hardware helps to drive down support and maintenance cost.

Second, there’s the question of power consumption. While the server-class hardware from Dell has a pretty serious power footprint, virtualization allows that footprint to be split between multiple machines that would otherwise each require their own chassis.

Third is physical space. While our data center isn’t really hurting in this area, I frankly don’t like clutter, and this was an opportunity to eliminate some. This is minor point in our case.

However, the biggest reason – by far – is disaster recovery (DR). I read an article recently that drove this point home with a couple of interesting statistics. First, that the majority of businesses that are forced to close for at least a month never reopen. Second, that DR costs are typically estimated to be 2x the cost of the underlying infrastructure.

Both stats make sense intuitively if you consider that most businesses have all of their value at one physical location. To the extent that there are digital assets that could be replicated offsite, that’s probably not being done. So if that’s all lost, due to a fire for example, the 2x factor means that the DR costs exceed those of just starting a new business. Scary.

Of course, the DR picture can look better when proper data back-up and recovery procedures are in place. To this point, the article suggests that VM images can be more readily archived and restored than physical drive images. I counter that by observing that data back-up and DR is hard, and expensive, no matter what. But once the proper infrastructure and education is in place, I do agree that VMs are a compelling option for these reasons.

Indeed, I think DR preparation will prove to be the first killer app for both virtualization and cloud computing for certain types of customer. While scalability is what drives some data centers to use virtualization, and is what’s driving some web application companies to experiment with cloud computing, DR is a concern for every business that has at least one computer. The industry will soon be at the point where it’s feasible, and even actually cheaper, to locate computing resources such as file storage and key LOB apps offsite even for non-technical businesses.

Thus, for all of those reasons, I migrated a physical 64-bit Windows Server 2003 file server over to Hyper-V. I essentially followed the steps in this blog post, including the use of VMware Converter and Vmdk2Vhd.

A few additional notes about that procedure: first, while the VMware tool ran fine on 64-bit Windows Server 2008, Vmdk2Vhd did not. Instead, I ran the latter on a 32-bit Vista workstation and just converted the files via the network. I’m sure I paid a non-negligible performance penalty, but it worked.

Which brings me to what I believe to be the main hurdle in performing this procedure against production servers: it takes a long time, end-to-end. File servers in particular are likely to hold lots of data. Converter must take an image of the data, while the machine is running, and copy it across the network. Our file server had two 150 GB disks. Together they were half-full. That took approximately 24 hours. Once that was done, I ran the two images through Vmdk2Vhd in parallel; that took around eight hours.

Those are unattended operations, of course, so it’s not like all that time is wasted. And while there’s probably an additional load on the target server during that time, it remains online and accessible. The main challenge I see is in synchronizing the new disk images with the production server until you finally cut over from the latter to the former in production. So, from the time Converter completes until the DNS/IP change-over to the VM, changes made to the disks on the original server will be lost unless you copy them manually (or, better, avoid them entirely) to the VM once it’s up.

Naturally, you want to follow the above steps carefully, and test the new VM before connecting it to the network. I wasn’t too scared, though: the nice thing about this procedure is that the existing server stays online until the very end. So if anything goes wrong bringing up the VM, you’ll have time to debug and try again. I got through the entire process without a single hitch, even though it was my first time.

Permalink | Comments (3)

I covet a solid-state laptop

December 18, 2008

Saw this (the ASUS EeePC 1000) in Fast Company this month.

The whole concept of hot having to deal with a power-sucking, loud, spinning harddrive is brilliant. Solid-state is more efficient (green tech, anyone?), lighter, and lasts longer.

This is one more significant step toward the convergence of cell phones and mobile computing.

Permalink | Comments (0)

The Securing Cyberspace for the 44th Presidency report is out. (If that title alone doesn’t get your blood pumping, you should probably skip the rest of this post …)

Interesting aside: one of the co-chairs of the commission that generated the report is Scott Charney, a security executive at Microsoft. This article also places him as a candidate for Obama’s Cyber Czar position!

Anyway, the report makes a couple of interesting recommendations, the summary of which starts on page 14. One is the creation of the Cyber Czar position, mentioned above. Another is to update government acquisition rules to require the use of “secure Internet protocols.”

There are also several recommendations regarding strong identity, consumer use of government-issued credentials online, and HSPD-12 (smart cards for Federal employees) specifically.

Permalink | Comments (0)

High-Performance Web Sites

December 16, 2008

Great article here about basic client-side web site performance problems. I happened to read it in this month’s hardcopy version of Communications of the ACM, and immediately thought, “wow, this is one of the best articles this magazine has ever run.” No joke.

What’s really cool is the analysis of how the top browsers, particularly IE and Firefox, parallelize content downloads and rendering, but that common page implementation mistakes get in the way. Simple stuff like where you should put style sheets (at the top) and where you should put scripts (at the bottom).

Also interesting is the analysis of the top 10 sites and how they’ve made many of the mistakes described in the article. For example, when cached content is available on the client, both Google and Live search are 100% server-bound (which is good).

In contrast, the MSN homepage is only 6% server-bound, implying that the browser is having to perform a lot of work before the page is finally rendered. Now, to some extent, that seems expected to me, since search engine results pages are quite barebones these days (not that they should be; they just are). Whereas the MSN homepage can’t afford to be barebones. But the question is, can a top 10 site really not afford to be 100% efficient?

The MSN page isn’t the only offender, either. Read the article …

Permalink | Comments (0)

JW Secure customer Blue Ridge Networks has just launched its AppGuard end-point security product.

AppGuard blocks bad application behavior at the system level. For example, this provides protection against zero-day attacks that may be missed by the user’s anti-malware suite, due to delays in availability of updated signature definitions.

Permalink | Comments (0)

For starters, see Chris Jackson’s blog. A recent post is here. That post is specifically about 64-bit issues, but it also gives a hint as to the general terminology to be aware of when it comes to application compatibility on Windows Vista.

Other resources:

Permalink | Comments (0)

Google Native Client

December 10, 2008

Google published a very interesting paper this week, entitled Native Client: A Sandbox for Portable, Untrusted x86 Native Code. The paper should be required reading for anyone who does security engineering or pentesting on Linux and Windows.

Native Client is essentially the Alpha version of a new browser plug-in model. It’s intended to expose high-performance processing capabilities to browser-based applications. For example, physics simulations, complex rendering, etc.

This blog post offers the Google security team’s take on the technology.

The coolest piece to me is the “reliable disassembly” and “inner sandbox” work that they’re doing – see the paper link, above. Given that the industry’s track record around browser security is quite bad, this is important work. Being able to run provably safe x86 in that environment is great research, although from a real-world implementation perspective, the devil is in the details.

The authors make a couple of comments about the number of lines of C code that would need to be tested and reviewed in order show that the trust boundaries in the Native Client system are in fact secure. In my experience, that’s misleading; the code is going to have bugs, and they’re going to get exploited.

Indeed, Microsoft has earned some nasty black eyes due to browser security mistakes; I hope Google knows what they’re getting into. I mean, is a new browser plug-in model really worth the risk in negative security PR?

To another point in the paper: within the security community, it’s a heavily debated whether open-source security review works. In other words, if a body of code isn’t widely deployed, such that it becomes an attractive target for glory seekers, and if qualified security experts aren’t being paid to examine it, by what basis can we assume that it will become secure just because it’s open source? I’d like to see a more thoroughly thought out strategy on how Native Client is going to be hardened.

It’ll be interesting to see if Google continues to invest in this area, and what their next steps will be. For example, the current Native Client system is 32-bit only – will a 64-bit version be forthcoming?

I also can’t wait to see what approach they take for system call moderation at the “outer sandbox” process boundary (see section 3.2 in the paper) on Windows and how well it coexists with the overall ecosystem (e.g. other API shims). The current outer sandbox prototype is apparently Linux-only, and uses ptrace.

Anyway, good job guys! Hope to see more …

Permalink | Comments (0)

I recently stumbled across this phenomenon: script-based web pages, easily discoverable via Google, that include PHP server debugging information. The basic ones start with the PHP _SERVER environment array and return all the values in that array as HTML to the client. That’s bad enough, because you’re exposing lots of server version and file system path information that could be useful to a hacker.

But there are pages out there that expose a lot more than that. For example, in addition to the typical web server information exposed in HTTP headers, you can also find kernel information, PHP configuration options, and version numbers for lots of other libraries like zlib, curl, xml, and mysql.

All a hacker needs to do is look at the version number of one of those libraries, look for security fixes that have been made since then, and then find exploits for those issues.

From a server administrator perspective, getting all of those components patched with the latest security fixes is notoriously difficult. Don’t make hackers lives easier by exposing that level of detail publicly. If you do need to host a PHP debugging script, make sure it’s not public, and/or delete it after you use it.

Permalink | Comments (0)
Newer Posts »