Other solutions are very poor for indexing github, SO, ubuntu forums and so on.
Crawling efforts should obviously be disttibuted but how to store data and access them for free?