Have you experimented with providing multiple samples for calibration taken from different perspectives? When I use a chessboard and OpenCV for camera calibration they recommend 10 or more image samples so the calibration algorithm can estimate a homography for each image and then iteratively find calibration parameters which minimize the reprojection error.