jest, enzyme - testing a method that returns jsx

I think you don't need to call

const renderLinksReturn = wrapper.instance().renderLinks(mockLinksData);

as it will be called when Navbar will be rendered.

Your solution is correct but in case you want some alternative robust ways to test it.

Since this test specifically tests for IndexLink and assumes that mockLinksData contains to = "/"

it('renderLinks should return a IndexLink when passed a link with to:\"/\"', () => {
    const wrapper = shallow(<Navbar linksData={mockLinksData}/>);

    // You can use both component name or component displayname
    // IndexLink or "IndexLink"
    expect(wrapper.find('IndexLink')).to.have.length(1);

    // In case you want to test whether indexLink has appropriate props or classes.
    const indexLink = wrapper.find(IndexLink).first();

   // Check whether indexLink has pass prop
   expect(indexLink.props().to).to.equal("/");

   // Check whether indexLink has correct classes set.
   expect(indexLink.hasClass('navbar-active-link')).to.equal(true);

  // Check whether indexLink displays the link test correctly
  expect(indexLink.find('navbar-link-text').text()).to.equal(mockLinksData.text);


 });

To build on top of @Nachiketha 's answer, that syntax won't work when what's returned is a fragment, this can be solved by wrapping the result in a div like:

const renderLinks = shallow(<div>
    {wrapper.instance().renderLinks(mockLinksData)
    </div>
)}

as suggested in this tread.

Faced similar issue where we were passing a jsx component to another component as a prop.

You can shallow render the returned jsx since it's like a valid React Function/Stateless Component. eg:

const renderLinks = shallow(wrapper.instance().renderLinks(mockLinksData))

And continue with your usual enzyme assertions.