How to add custom validation rule in javascript using magento 2?

You can add a new validation rule with the data-validate attribute of the HTML tag using Magento.

By Default Magento mage/validation.js file used to set out-of-the-box custom rules for the Magento. Most of the times custom rules are used in the input tags. though you can apply to form tag, select tag as well as any HTML tag.

Syntax:
$.validator.addMethod(argument1, argument2, argument3) used to define custom validation rule in Magento.

argument1: Custom Validation rule name
argument2: Callback function to check logic using the function, the result will be always true/false.
argument3: Rule Error Message to be displayed on the failed validation.

There are two ways you can define your custom rule.
1. Using mixins of mage/validation file,
Create a requirejs-config.js file in your module,

var config = {
    config: {
        mixins: {
            'mage/validation': {
                'Rbj_CustomRule/js/validation': true
            }
        }
    }
}

Here we are creating mixins for the mage/validation to add custom rules in our file.

Create app/code/Rbj/CustomRule/view/frontend/web/js/validation.js file.

define(['jquery'], function ($) {
    'use strict'

    return function () {
        $.validator.addMethod(
            'custom-rule-name',
            function (value) {
                if (value != true) {
                    // logic to check validation.
                    return false;
                }
                return true;
            },
            $.mage.__('Your Validation failed. Please enter correct data.')
        );
    }
});

Now In your template file,

<input type="text"
       name="rakesh"
       value=""
       class="input-text"
       data-validate="{required: true, 'custom-rule-name': true}" />

You can add using data-validate=”{required: true, ‘custom-rule-name’: true}” in your input tag.
Clear Cache and check your input field.

2) Using ‘mage/validation’ as a dependency to the javascript file,

define([
    'jquery',
    'mage/validation'
], function ($) {
    $.validator.addMethod(
        'custom-rule-name',
        function (value) {
            if (value != true) {
                return false;
            }
            return true;
        },
        $.mage.__('Your Validation failed. Please enter correct data.')
    );
});

You can replace ‘custom-rule-name’ with your rule name and message to be displayed.

3) Using ‘jquery/validate’ as Dependency to the Define() method in javascript,

define([
    'jquery',
    'mage/translate',
    'jquery/validate'
], function ($, $t) {
    'use strict';

    $.validator.addMethod(
        'custom-rule-name', 
         function (value) {
            //your logic
	    // always return boolean value
        },
        $t('Your Validation failed. Please enter correct data.');
});