No, you cannot refuse an inspection. The insurance company has the right to inspect your home to determine the extent of the damage and the cost of repairs.
If you’re not home for the inspection, the insurance company will reschedule the inspection for another time.