HACKER Q&A
📣 3big5ram

Essential readings on dealing with data that does not fit in RAM?


When developing data processing systems like for example database engines or some massive simulations programs may run into problems like RAM being not enough to store some portions of result data or datasets that program has to process are too big to load into RAM at once leaving no other choice than introducing some mechanism of loading/storing parts of data from hard drive before/during/after processing (like for example sorting, merging data structures together etc). My question is as follows: is there some good literature on algorithms for dealing with such cases? Would be extra cool if there are some publications that take notice of things like CPU cache lines and/or RAM related stuff like multichannel memories, NUMA etc. If there is stuff that discusses even multiport RAM I would be interested in that stuff too.

Thanks in advance to everyone recommending any readings.


  👤 enduku Accepted Answer ✓