How to load a phtml file only for homepage? (Magento 2)

As per as, Magento2, default.xml is call at every page.If you add this code at default.xml then it would be automatically call every pages of your current theme.

If you know about Magento 1.x handler concept then you can understand that default is a handle which is called at every pages.In Magento 2.X,every handler is individual layout files.

So if want to add a phtml file only at home page then you need to add that code at cms_index_index.xml (/app/design/frontend/name/themename/Magento_Cms/layout/cms_index_index.xml) layout file because of cms_index_index is handler which call only called at home

You can use xml to do this task
Inside cms_index_index.xml put xml like this

<referenceContainer name="content">
        <container name="block.container" htmlTag="div" htmlId="slider.container" htmlClass="block-home-container" before="">
            <block class="Magento\Framework\View\Element\Template" name="block.banner" as="block.custom.cms" template="Magento_Theme::block.phtml" after="-" />
        </container>
</referenceContainer>

Create xml file name with /app/design/frontend/Vendor/theme/Magento_Theme/layout/cms_index_index.xml

<?xml version="1.0"?>
<page xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" layout="1column" xsi:noNamespaceSchemaLocation="urn:magento:framework:View/Layout/etc/page_configuration.xsd">
   <referenceContainer name="page.top">
     <block class="Magento\Framework\View\Element\Template" name="startphoto" template="Magento_Theme::html/startphotos.phtml">
        <arguments>
            <argument name="section" xsi:type="string">homepage</argument>
            <argument name="position" xsi:type="number">0</argument>
        </arguments>
     </block>
  </referenceContainer>         
</page>