How to render HTML with AJAX in Magento 2

I would also go the way 2 and, indeed, you actually can render "pure" HTML via AJAX without the head, body, css and so on.

The trick is to:

• tell your controller to instanciate a Response that is of type \Magento\Framework\View\Result\Layout rather than \Magento\Framework\View\Result\Page
• use a layout XML file with a root node that is <layout...>...</layout> rather than <page...>...</page>

Here is a very simple implementation.

The controller

<?php    
namespace Namespace\Module\Controller\Index;

use Magento\Framework\App\Action\Action;
use Magento\Framework\App\ResponseInterface;
use Magento\Framework\Controller\ResultFactory;

class Index extends Action
{
    /**
     * Dispatch request
     *
     * @return \Magento\Framework\Controller\ResultInterface|ResponseInterface
     * @throws \Magento\Framework\Exception\NotFoundException
     */
    public function execute()
    {
        return $this->resultFactory->create(ResultFactory::TYPE_LAYOUT);
    }
}

The layout

<?xml version="1.0"?>
<layout xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:framework:View/Layout/etc/layout_generic.xsd">
    <container name="root">
        <block class="Namespace\Module\Block\Some\Block" name="namespace_module.some_block" />
    </container>
</layout>

Example on Github

See this example module: https://github.com/herveguetin/Herve_AjaxLayout_M2

This module generates this:

Out of the box, Magento does not use any of those methods to render HTML via AJAX.

From what I've seen, everytime such thing needs to be done, JSON is used to transport the result.

Example from the Magento/Checkout/Controller/Cart/Add :

$this->getResponse()->representJson(
    $this->_objectManager->get('Magento\Framework\Json\Helper\Data')->jsonEncode($result)
);

Then Magento 2 uses a new mechanism called sections, to handle the data on the frontend and update the specific blocks that need to be updated, you can learn more about sections in this Q&A: https://magento.stackexchange.com/a/143381/2380

EDIT regarding the second part of my answer: as stated by Max in the comment, sections are only used with customer specific data and using this functionality instead of every AJAX call is not the right solution.

In my example I can not use sections because it is not customer data and it is not after a PUT / POST action but using Raphael at Digital Pianism answer I figured out how Magento render sections.

If we take the example of cart section it use the method \Magento\Customer\CustomerData\SectionPool::getSectionDataByNames to retrieve data from sections. This lead us to \Magento\Checkout\CustomerData\Cart::getSectionData with a single array containing areas of the section, including $this->layout->createBlock('Magento\Catalog\Block\ShortcutButtons')->toHtml()

Depending on that here is the final Controller class:

<?php

namespace Foo\Bar\Controller\Popin;

use Magento\Framework\App\Action\Action;
use Magento\Framework\App\Action\Context;
use Magento\Framework\Controller\Result\JsonFactory;
use Magento\Framework\Data\Form\FormKey\Validator;
use Psr\Log\LoggerInterface;

/**
 * Class Content
 */
class Content extends Action
{

    /**
     * @var LoggerInterface $logger
     */
    private $logger;
    /**
     * @var Validator $formKeyValidator
     */
    private $formKeyValidator;
    /**
     * @var JsonFactory $resultJsonFactory
     */
    private $resultJsonFactory;

    /**
     * Content constructor.
     *
     * @param Context $context
     * @param LoggerInterface $logger
     * @param Validator $formKeyValidator
     * @param JsonFactory $resultJsonFactory
     */
    public function __construct(
        Context $context,
        LoggerInterface $logger,
        Validator $formKeyValidator,
        JsonFactory $resultJsonFactory
    ) {
        $this->logger            = $logger;
        $this->formKeyValidator  = $formKeyValidator;
        $this->resultJsonFactory = $resultJsonFactory;
        parent::__construct($context);
    }

    /**
     *
     */
    public function execute()
    {
        if (!$this->formKeyValidator->validate($this->getRequest())) {
            return $this->resultRedirectFactory->create()->setPath('checkout/cart/');
        }

        /** @var \Magento\Framework\Controller\Result\Json $resultJson */
        $resultJson = $this->resultJsonFactory->create();

        try {
            /** @var \Magento\Framework\View\Layout $layout */
            $layout = $this->_view->getLayout();
            /** @var \Foo\Bar\Block\Popin\Content $block */
            $block = $layout->createBlock(\Foo\Bar\Block\Popin\Content::class);
            /** @var array $response */
            $response = [
                'content' => $block->toHtml(),
            ];
        } catch (\Exception $exception) {
            $resultJson->setStatusHeader(
                \Zend\Http\Response::STATUS_CODE_400,
                \Zend\Http\AbstractMessage::VERSION_11,
                'Bad Request'
            );
            /** @var array $response */
            $response = [
                'message' => __('An error occurred')
            ];
            $this->logger->critical($exception);
        }

        return $resultJson->setData($response);
    }
}