XBee dropping packets and actual data rate lower than expected

You can [reduce to zero the dropped packages][1] by assigning the correct destination address before starting transmission.

whats the signal strength and speed of the wireless link look like? Check the XBee API docs you should be able to access this information. What antennas are you using?

Zigbee's raw data rate is only 250kbit/sec in the 2.4Ghz band and it is a very high overhead protocol. With near perfect signal strength and encryption enabled you should only expect ~20-25kbit/sec max data throughput without customizing the stack, a bit more without encryption. Zigbee's protocol really only supports sending data that fits in a single packet, which off the top of my head is something like 100 bytes. If your sending a stream of data the application layer has to break that data into packets, and include additional information in the data space of the packet so that it can be reassembled. This process can be quite slow and cut further into the data throughput.

Digi's digimesh stack is a bit faster as it cuts overhead and allows larger packets.

Not sure what your intended final application is here, but Zigbee is not at all designed for streaming data. Its for sending small bits of information, sensor readings, instructions, etc that fit into a single packet. You very well may have just chosen the wrong protocol for your application.