The localization system is one of the key components of autonomous vehicles. Widely-used GNSS (low-cost) alone can not meet the centimeter accuracy and is unstable in urban environment. Typically LiDAR-based localization system is very computationally demanding due to the usage of the dense point cloud map. In this paper, we propose a novel lightweight LiDAR-based localization system for autonomous vehicle in this paper. The proposed system only relies on lightweight poles and curbs landmark map, which is highly robust and efficient compared to other localization systems. Poles and curbs are selected as landmarks because of their commonality and stability. We novelly propose a Branch-and-Bound (BnB)-based global optimization method to tackle the data association problem of poles. Motion decoupling is adopted to decouple translation and rotation to improve the efficiency of the BnB-based algorithm. Besides, we propose a new local grid map-based representation for curbs to make better use of curb information. Cost functions for pole and curb are defined respectively and then fused for the subsequent non-linear optimization method to obtain the vehicle location. Experiments on KITTI dataset and our self-collected dataset demonstrate the efficiency and accuracy of our system.