Creating the root Vue instance inside of a Vue file

What you are describing is done in a pre-compilation step through Webpack and Vue Loader. The Vue compiler doesn't actually parse Single File Components. What the Vue compiler can parse is templates provided in a component’s options object. So if you provide a template option in your componentConfig object your example will work. Otherwise you'll have to go through the pre-compilation step with Webpack and Vue Loader to parse a Single File Component's template. To do that you'll have to conform to the SFC structure defined in the spec. Here's an excerpt ..

Template

• Each *.vue file can contain at most one <template> block at a time.
• Contents will be extracted and passed on to vue-template-compiler and pre-compiled into JavaScript render functions, and finally injected into the exported component in the <script> section.

Script

• Each *.vue file can contain at most one block at a time.
• The script is executed as an ES Module.
• The default export should be a Vue.js component options object. Exporting an extended constructor created by Vue.extend() is also supported, but a plain object is preferred.
• Any webpack rules that match against .js files (or the extension specified by the lang attribute) will be applied to contents in the <script> block as well.

---

To make your specific example work You can re-write main.js file like this ..

const MyComponent = require("./MyComponent.vue");

const el = ".container";
const data = {
  name: "Jess"
};

MyComponent.create(el, data);

And your MyComponent.vue file (This could just as well be a js file as @Ferrybig mentioned below) ..

<script>
const Vue = require('vue');

function create(el, data) {
  const componentConfig = {
    el,
    name: "my-component",
    data,
    template: `<div>Hello {{ name }}!</div>`
  };
  const vm = new Vue(componentConfig);
  return vm;
}

module.exports = { create };
</script>

See this CodeSandbox

Or if you prefer render functions your MyComponent.vue will look like this ..

<script>
const Vue = require('vue');

function create(el, data) {
  const componentConfig = {
    el,
    name: "my-component",
    data,
    render(h) { return h('div', 'Hello ' + data.name) } 
  };
  const vm = new Vue(componentConfig);
  return vm;
}

module.exports = { create };
</script>

CodeSandbox

---

One last thing to keep in mind: In any component you can use either a template or a render function, but not both like you do in your example. This is because one of them will override the other. For example, see the JSFiddle Vue boilerplate and notice how when you add a render function the template gets overridden. This would explain that error you were getting. The render function took precedence, but you fed it a component's options object that provides no template to be rendered.

You are really close to a working solution, but you are missing just some "glue" parts to combine everything together:

<template>
    <div>Hello {{ name }}!</div>
</template>
<script>
    import HelloWorld from "./components/HelloWorld";
    import Vue from "vue";

    const Component = {
        // Add data here that normally goes into the "export default" section
        name: "my-component",
        props: {
            name: String,
        },
        data() {
            return {
            };
        },

    };
    Component.create = function(el, props) {
        const vm = new Vue({
            el,
            render(h) {
                return h(Component, { props });
            },
        });
        vm.$mount();
        return vm;
    };
    export default Component;
</script>

This can then be used as follows in other files:

import App from "./App";
App.create("#app", {
    name: 'Hello World!',
});

Example on codesandbox: https://codesandbox.io/s/m61klzlwy