OCCT Viewer doesn't implement such selection for the moment - either it picks by a ray/pixel with results sorted along the picking ray (AIS_InteractiveContext::SelectPoint()), or it picks everything that fits into frustum (AIS_InteractiveContext::SelectRectangle() / ::SelectPolygon()). Detection mechanism works independently from Visualization, so that it cannot (easily) detect if particular entity is behind another object or not.
I can imagine some ways to implement desired filtering of points at application level. The most brute-force approach could be first picking all vertices within frustum (AIS_InteractiveContext::SelectRectangle() / SelectMgr_ViewerSelector::Pick()), then re-perform picking by ray from eye to each detected point (using SelectMgr_ViewerSelector::Pick() taking gp_Ax1 on input) to reject vertices having surfaces in front (by comparing depth/distance or other means). Cannot say what kind of performance this approach could show, but it might be not that bad.