The only major difference was a newer version of the Apache Web Server -- 2.2.21 vs 2.2.16.
Unfortunately, not long after the switch, reports started coming in that most attachments were not uploading. Only very small ones (less than 100K) seemed to work.
Since there were no recent changes to our attachment code, I started digging through logs to discover the source of the problem.
None of the usual error logs reported seemed to contain anything useful.
Finally I worked my down down to the root Apache error log and discovered this little gem:
[warn] [client ...] mod_fcgid: HTTP request length 137881 (so far) exceeds MaxRequestLen (131072), referer:
A nondescript little warning that thankfully led me to this post which solved the problem.
This person also seemed to have had the same problem.
The icing on the cake was the little warning at http://httpd.apache.org/mod_fcgid/mod/mod_fcgid.html#fcgidmaxrequestlen, which reads:
Warning: Before 2.3.6, this defaulted to 1GB. Most users of earlier versions should use this directive to set a more reasonable limit.
My question is WHY!?! Why did they reduce a 1GB default limit to 128K? Surely someone must have realized that havoc that might wreak, no?
In any case, if you encounter this problem, simply add the following to your Apache config:
IfModule mod_fcgid c FcgidMaxRequestLen IfModule
and you should be good to go again.