In Java how instance of and type cast(i.e (ClassName)) works on proxy object?

Java isn't allowing casting from a proxy to a concrete class. JDK proxies (java.lang.reflect.Proxy) are only proxies of an interface. The resulting proxy is of type ProxyX (X being a number), and if you try to cast it to any class you will get ClassCastException

Therefore your 2nd and 3rd questions aren't relevant - the proxy isn't backed by a concrete class. To achieve this, you can use other proxying mechanisms - CGLIB or javassist. They use ynamic subclassing, and so all protected (and above) fields and methods are accessible to the subclass (proxy).

From the API javadocs for java.lang.reflect.InvocationHandler:

InvocationHandler is the interface implemented by the invocation handler of a proxy instance.

The dynamic proxy implements the interface, but uses the handler (OriginalClass) to provide the base implementations of the methods.

To answer your questions:

1. The compiler will let you cast as long as it doesn't have enough information to be certain that the cast cannot succeed. The runtime behaviour of casting and instanceof tests for dynamic proxies is described in the javadoc for java.lang.reflect.Proxy. Casts and instanceof tests will succeed if used with interfaces, but not if used with classes.
2. You cannot access any attributes using the dynamic proxy because it implements the interface, it does not extend the handler class.
3. You cannot access any methods not declared in the interface using the dynamic proxy because it implements the interface, it does not extend the handler class.

Inside the implementation of the dynamic proxy (e.g. in the implementation of the invoke(...) method) you can access the members of the handler using reflection.

Here's some test code that I used to check my answer:

// package ...;

import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;

import junit.framework.Assert;

import org.junit.Test;

public class TestDynamicProxy
{
    @Test
    public void testCast() throws Exception {
        Foo foo = (Foo) TestProxy.newInstance(new FooImpl());
        foo.bar(null);

        System.out.println("Class: " + foo.getClass());
        System.out.println("Interfaces: " + foo.getClass().getInterfaces());

        Assert.assertNotNull(foo);
        Assert.assertTrue(foo instanceof Foo);
        Assert.assertFalse(foo instanceof FooImpl);
    }
}

interface Foo
{
    Object bar(Object obj) throws Exception;
}

class FooImpl implements Foo
{
    public Object bar(Object obj) throws Exception {
        return null;
    }
}

class TestProxy implements java.lang.reflect.InvocationHandler
{
    private final Object obj;

    public static Object newInstance(Object obj) {
        return java.lang.reflect.Proxy.newProxyInstance(obj.getClass().getClassLoader(), obj.getClass().getInterfaces(), new TestProxy(obj));
    }

    private TestProxy(Object obj) {
        this.obj = obj;
    }

    public Object invoke(Object proxy, Method m, Object[] args) throws Throwable {
        Object result;

        try {
            result = m.invoke(obj, args);
        }
        catch (InvocationTargetException e) {
            throw e.getTargetException();
        }
        catch (Exception e) {
            throw new RuntimeException("unexpected invocation exception: " + e.getMessage());
        }

        return result;
    }
}

This article has a lot of useful information and example code.