Even with preference alignment, LLMs can be enticed into harmful behavior via adversarial prompts 😈.

🚨 Breaking: our theoretical findings confirm:
LLM alignment is fundamentally limited!

More details, on framework, statistical bounds and phenomenal defense results 👇🏻