(Take with some grains of salt: never used FreeBSD or Solaris shells myself and am not a ts-admin. Also, this is a cell phone and I shouldn't even be awake now!)

Also, what do you lose with FreeBSD?

so far I can think of:
* zfs dedupe
* there's no longer a single "goto" company to get support from (guessing...) so you lose that annual cost but it may be non trivial to find the right hacker to find/fix a problem in an emergency. of course much of that support is likely to come for free (a guess)
* you're still a different platform from the rest of wikimedia (basically the foundation) and so lose economy of scale
* you mentioned SGE would still be available but it may be unmaintained (last I checked its enwp article, it said oracle was close sourcing it) just to keep in mind, I wouldn't stay just for SGE

things you lose with linux vs. FreeBSD:
* zfs is gone and (assuming lvm2 + traditional RAID vs. zfs) generally thin storage provisioning and snapshotting is much more limited and wasteful and fragile/more room for error. also most storage expansions will end up w/ transition periods that have *no* redundancy.
* zones/jails

I haven't kept up with the status in the last ~8 months but debian-kfreebsd may have matured enough to warrant a look. (See #debian-kbsd on oftc)

-Jeremy