Does the number of visits matters for SEO?

Search engines have no way of knowing what traffic your website gets so it can't be used as a ranking metric in their algorithm. (Google has clearly stated that Google Analytics data is not used in their ranking algorithm).

Even if they did, the number of visits would not be a good judge of relevance as it is easy to artificially inflate your number of visitors. So even if search engines knew how many visitors your site had it would not be a very useful metric to them. (Now if they had access to your site analytics they could use that to determine the quality of your site visits which could possibly be used to determine quality of content but that is currently not happening).

Google can measure visitors if the website is a click through on their search engine and this is across the board with all the other search engines. Google has never directly confirmed that trends of clicks actually matters or not, through it would make logical sense that this would be true.

So other than having direct access to your server there is no way to reward for direct visitors since they can't detect it. There is however a myth that Google can use Google Analytics data since they changed their privacy agreement which means this data can be used in conjunction with their other services, while this may be true its never been confirmed. So this is a highly debatable topic because even if you don't have Google Analytics the vast majority do so I know you said you don't have links but if you had they can potentially see this too as an exit point.

You should regard Direct visitors good as it can potentially indirectly improve your SEO by gaining links.

Search Engine may know about visits to orphaned (i.e. no inbound links) websites based on users who may have a toolbar installed that is reporting browsing activity back to the search engine. Whether this is used as a ranking signal is debatable.

Also if you use a tool such as Google Analytics then Google in this case will certainly know a great deal about your website users. Again it's very unlikely that they will be using this as any basis to rank your site, which I think is at the crux of your question.

Inbound links and good content pretty much cover 90% of the ranking outcome for any website. Nothing has changed there for years!