What is your record number of file descriptors on Linux?
If you bump up the ulimit soft and hard, and even re-compile the kernel and set NR_OPEN > 600000 how high can you get it before aws networking or some other limitation kicks in? Just one single machine (could even be bare metal if you want) but it can have a very high amount of memory and cpus. Has anyone done over 1,000,000?
I don't remember the exact number but there was a bug in some java code that was leaving orphaned FD's. It was in the millions. This was on machines that had between 512GB and 3TB of ram and did not seem to affect them, just made debugging things harder. I logged the issue in Jira and nobody cared. They restarted the code and servers enough that it did not seem to affect anything.