You can delete the specific customer address by address id using the AddressRepositoryInterface interface.
Magento\Customer\Api\AddressRepositoryInterface contains the deleteById($addressId) function with pass first argument as address id to fetch specific customer address data.