How can I tell if a search was successful in vimscript?

There are various approaches, depending on the context.

:if there's a ( on the current line do foo :endif

You could use search(), as per @Kent's answer. It supports {stopline} argument, to avoid that it goes beyond the current line (which you can pass via line('.')). But it only searches in one direction (either forward or backward), so you'd have to position the cursor.

So it sounds like if getline('.') =~ '(' is a better test. It does a regular expression comparison of the current line with (. You could also use match() instead (look up any function via :help for the full API documentation and examples BTW), or a non-regexp stridx() (which might be faster, but also is less clear to read).

:if /searchtarget succeeds do bar :endif

Again, this sounds like a use for search(), which repositions the cursor on a match like /search. But you could also use the latter (with :normal), and check for a jump by comparing the cursor positions (obtained via getpos('.')) before and after the command.

Vim has search() function. :h search( to read the function description.

If a match was found, the func will return the line number, otherwise return 0. You can do your logic based on the return value of the function.

To check if some pattern in current line, you can use the search() function too, also you can split() the text of current line, to see if you got the result list with more than 2 elements.