How to add custom validation rule in javascript using magento 2?

You can add a new validation rule with the data-validate attribute of the HTML tag using Magento.

By Default Magento mage/validation.js file used to set out-of-the-box custom rules for the Magento. Most of the times custom rules are used in the input tags. though you can apply to form tag, select tag as well as any HTML tag.

Syntax:
$.validator.addMethod(argument1, argument2, argument3) used to define custom validation rule in Magento.

argument1: Custom Validation rule name
argument2: Callback function to check logic using the function, the result will be always true/false.
argument3: Rule Error Message to be displayed on the failed validation.

There are two ways you can define your custom rule.
1. Using mixins of mage/validation file,
Create a requirejs-config.js file in your module,

Here we are creating mixins for the mage/validation to add custom rules in our file.

Create app/code/Rbj/CustomRule/view/frontend/web/js/validation.js file.

Now In your template file,

You can add using data-validate=”{required: true, ‘custom-rule-name’: true}” in your input tag.
Clear Cache and check your input field.

2) Using ‘mage/validation’ as a dependency to the javascript file,

You can replace ‘custom-rule-name’ with your rule name and message to be displayed.

3) Using ‘jquery/validate’ as Dependency to the Define() method in javascript,