You can challenge your neighbor if you know the law. When I bought my house many years ago, I did not know that I lost 77 SQ/FT at the back of the property line. Finally, I took down the old steel fence (built by a Vietnamese, first owner) and I added 77 SQ/FT when I built the new wood fence based upon the land survey.