Can CHECK constraints act like if else?

Absolutely, you can do this. See this sqlfiddle.

However, you need to make sure you bracket your logic properly. You should never mix ANDs and ORs in the same bracketing scope. So:

(col1 NOT NULL OR col2 NOT NULL AND col3 NULL)

Needs to become:

((col1 NOT NULL OR col2 NOT NULL) AND col3 NULL)

Or:

(col1 NOT NULL OR (col2 NOT NULL AND col3 NULL))

Depending on your intent.

I would say create a UDF like below

create FUNCTION dbo.fn_check_val
  (@col1 int , @col2 int , @col3 int)
RETURNS bit
AS
BEGIN
    declare @toRet bit
    IF(@col1 is Not null OR @col2 is NOT NULL)
    Begin
        if(@col3 is null)
        Begin
            Set @toRet = 1
        End
        Else
        Begin
            Set @toRet = 0
        End
    End
    Else
    if(@col3 is not null)
    Begin
        Set @toRet = 1
    End
    Else
    Begin
        Set @toRet = 0
    End
return @toRet
END

and then add following check statement in your table

([dbo].[fn_check_val]([col1],[col2],[col3])=(1))

Just be careful not to make mistake with brackets.

CREATE TABLE Test1 (col1 INT, col2 INT, col3 INT);


ALTER TABLE Test1 
ADD CONSTRAINT CHK1
CHECK  (((col1 IS NOT NULL OR col2 IS NOT NULL) AND col3 IS NULL) OR 
        ((col1 IS NULL AND col2 IS NULL) AND col3 IS NOT NULL))



INSERT INTO Test1 VALUES (1,1,1); --fail
INSERT INTO Test1 VALUES (1,1,NULL); --good
INSERT INTO Test1 VALUES (1,NULL,NULL); --good
INSERT INTO Test1 VALUES (1,NULL,1); --fail
INSERT INTO Test1 VALUES (NULL,NULL,1); --good