The Case Against LLMs as Rerankers