Towards Robust And Lightweight Vision-Based Pedestrian Trajectory Prediction