how to prevent postback by javascript

It's not working since you have a script error in your javascript.

<script type="text/javascript">
    function doSubmit() 
    {
        //I've removed the first equal sign
        var ansLength = oDOM.body.innerText.trim().length; 
        if (ansLength == 0 && smielyPresent == -1) //typo on smielyPresent ?
        {
            alert("u can't submit blank answer")
            return false;
        }
    }
</script>

try this

OnClientClick="doSubmit(this); return false;"