Secondary and Arbiter stuck in Startup
From "BOSPROD9", try to connect with the mongoshell to the other serves:
$ mongo --host 10.235.96.12 --port 27017
$ mongo --host 10.235.96.13 --port 27017
(Telnet is not the same.) If this doesn't work, it might be firewall or BindIP.
Check bind_ip (should be 0.0.0.0, change in mongodb.conf is it's 127.0.0.1):
$ netstat -nap | grep :27017 | grep LISTEN
tcp 0 0 0.0.0.0:27018 0.0.0.0:* LISTEN -
Try to look at the log-files on 10.235.96.12 and 10.235.96.13, why they are stuck. Did they receive the configuration?
Try to reconfigure this way:
mongo> var cfg = {_id:"ReplicaSet1",members:[{_id:0, host:"BOSPROD9:27017"},{_id:1, host:"10.235.96.12:27017"},{_id:2, host:"10.235.96.13:27017",arbiterOnly:true}]};
mongo> rs.reconfig(cfg);
Solution:
Use hostnames instead of ip's: give all servers a hostname, update the hosts-files and use hostnames in the rs-configuration. It seems that mongodb advices not to use ip addresses, but hostnames.