The Alignment Problem Under Partial Observability