You can remove specific attribute set from the system by given id in Magento 2.
If you want to delete extra attribute set from the magento admin panel, You can safely remove it by given id in the deleteById($attributeSetId) from the AttributeSetRepository Interface.