core: Make sure FD_SET is not used with an out of range fd.