it seems crazy for LWN to spend 100's of hours of valuable time and resources and to eventually still capitulate to the "ignore robots.txt" crowd if just using javascript could solve the problem. Does ...