Anyonic symmetry and twist defects