Abstract: Point cloud analysis is challenging because of the unordered and irregular data structure of point clouds. To describe geometric information in point clouds, existing methods mainly use convolution, graph, and attention operations to construct sophisticated local aggregation operators. These operators work well in extracting local information but bring unfavorable inference latency due to high computation complexity. To solve the above problem, this paper presents a novel point-voxel based geometry-adaptive network (PVGANet), which combines multiple representations of point and voxel to describe the point cloud from different granularities and can obtain features of different scales effectively. To extract fine-grained geometric features, we design the position-adaptive pooling operator, which uses point pairs’ relative position and feature similarity to weight and aggregate the point features at local areas of point clouds. To extract coarse-grained local features, we design a depth-wise convolution operator, which conducts the depth-wise convolution on voxel grids. With an easy addition, fine-grained geometric and coarse-grained local features can be fused, and we can use the geometry-adaptive fused features to complete the efficient shape analysis of point clouds, such as shape classification and part segmentation. Extensive experiments on ModelNet40, ScanObjectNN, and ShapeNet Part benchmarks demonstrate that our PVGANet achieves competitive performance compared with the related methods.