Using the Batch API with AJAX?

It is possible without hacking core. It needs a bit of fiddling, but you can basically just use Drupal.ProgressBar for an existing batch process, providing your own updateCallback to control its behaviour.

I have successfully done this in every version of Drupal since Drupal 7, but can't share my code here unfortunately.

One example I found for Drupal 8 explains the gist of it, although it's not explicitly covering the modal part of the question: https://www.hashbangcode.com/article/drupal-8-running-batch-through-ajax-request

I'll share a slightly modified (and untested) version of the code example from the linked article:

(function ($, Drupal) {
  'use strict';

  Drupal.behaviors.account = {
    attach: function attach(context, settings) {
      var progressBar = void 0;

      function updateCallback(progress, status, pb) {
        $('#updateprogress').html(progress + '%');
        if (progress === '100') {
          pb.stopMonitoring();
          // And do whatever suits your needs, redirecting or updating some DOM
          // elements
        }
      }

      function errorCallback(pb) {
      }

      $.ajax({
        url: Drupal.url('account/loading/ajax'),
        type: 'POST',
        contentType: 'application/json; charset=utf-8',
        dataType: 'json',
        success: function success(value) {
          progressBar = new Drupal.ProgressBar('updateprogress', updateCallback, 'POST', errorCallback);
          progressBar.startMonitoring(BATCH_URL + '&op=do', 10);
        }
      });
    }
  };
})(jQuery, Drupal);

BATCH_URL is the url that you get after setting up the batch process in PHP. I would also end the process with a BATCH_URL + '&op=finished' to do cleanup on the server, evaluate the process or whatever makes sense in the specific context.

You can also extend the Drupal.ProgressBar.prototype object to further modify the behaviour of it and adapt it your needs, e.g. by overriding its setProgress to do more fine grained updates or provide server side generated messages.

Steps To Run batch through jquery ajax request:

1. Create controller in which batch API code ‘ll be added

Snippet code:

public function updateNodeData(Request $request, $nojs = 'nojs') {
    $method = $request->server->get('REQUEST_METHOD');
    $ajaxResponse = new AjaxResponse();
    // Add logic to check whether request to batch controller by
    // post method or not so that batch process not redirect to batch
    // window itself (i.e. by default core functionality)
    if ($nojs == 'ajax' && $method == 'POST') {
      $all_values = $request->request->all();
      $fields = $all_values['fields'];

      $node_storage = $this->entityTypeManager()->getStorage('node');
      $query = $node_storage->getQuery();

      $ids = $query->condition('type', $fields['content_type'])
        ->accessCheck(FALSE)
        ->execute();

      $batch_builder = new BatchBuilder();
      $batch_builder
        ->setTitle($this->t('Updating Data ..'))
        ->setFinishCallback([
          '\Drupal\batch_example_without_redirect\NodeData',
          'exportFinished',
        ])
        ->setInitMessage($this->t('Updating Node.'))
        ->setErrorMessage($this->t('The process has encountered an error.'));

      $ids_chunk = array_chunk($ids, 100);
      foreach ($ids_chunk as $id) {
        $batch_builder->addOperation([
          '\Drupal\batch_example_without_redirect\NodeData',
          'processNodeData',
        ],
        [
          $id,
        ]);
      }

      batch_set($batch_builder->toArray());

      // Get batch.
      $batch =& batch_get();

      $batch['progressive'] = TRUE;

      // Create the batch_process() to initiate batch process.
      $response = batch_process();

      $batch_id = $batch['id'] ?? '0';

      // Return the response to the ajax output.
      return $ajaxResponse->addCommand(new BaseCommand($batch_id, $response->getTargetUrl()));

    }

    return $ajaxResponse;

}

2. Then call the controller/endpoint through jquery ajax request in which batch API code added so that batch process ‘ll be initiate

Snippet code:

  // Call batch controller by ajax & post method.
        const url = 'update-node-data/ajax';
        $.ajax({
          url: Drupal.url(url),
          type: 'POST',
          data: {
            'fields': settings.user_submit_val.fields,
          },
          dataType: 'json',
          success: function success(value) {
            batch_id = value[0].command;
            // Display batch progress
            progressBar = new Drupal.ProgressBar('updateprogress', updateCallback, 'POST');
            progressBar.setProgress(0, 'Updating Data');
            progressBar.startMonitoring(value[0].data + '&op=do', 10);
          }
        });

3. After batch completed its execution :

Add check if progress == 100 that mean if all the items in batch processed, then call the batch finished callback endpoint through the jquery ajax request

Snippet code:

function updateCallback(progress, status, pb) {
    $('#updateprogress').html(progress + '%');
      if (progress === '100') {
        pb.stopMonitoring();
        // Call the finishCallback()
        $.ajax({
          url: Drupal.url('batch?id=' + batch_id + '&op=finished'),
          type: 'POST',
          contentType: 'application/json; charset=utf-8',
          dataType: 'json',
          success: function success(val) {
            window.location.reload();
          }
        });
      }
  }

Note: if you ‘ll skip above mentioned step 3 : then code added in batch finished callback ‘ll never execute.

If you want to pass some dynamic user selected values to batch then you need to add some additional steps along with above mentioned steps:

1. Create custom form & on submit form, call the ajax callback which ‘ll send user submit data from form to jquery.

2. Then in the attached jquery file get the user submit form values & pass these values to the batch controller by jquery ajax request.

3. Fetch the user submit form values in batch controller & use it as per logic.

Note: For complete example you can download demo module from here to see working example of it for the reference.