Using ping google.com to test connection

Unfortunately you really have not provided enough information to work out where the problem is. To answer as best as I possibly can with the limited information provided:

1. If my experiences are anything to go by, pinging Google is normally quite a good bet, as they design their network to be as fast as possible. Also as ICMP is prioritized, evening peak probably does not make a significant difference - particularly in terms of packet loss - which I'd argue should be 0.

2. Google is a good destination, but to get a better idea of what is going on you may want to additionally try pinging your gateway, and if they allow it, your providers DNS, mail or web server. This will help show where the packet loss is creeping in. Realistically though, at the level of packet loss you are seeing, look at downloading MTR (or WinMTR) and running that at peek to get a better idea of where packet loss is coming in.

3. Subjectively, 5% packet loss is at the top end of acceptable for a Wifi based network - assuming you are not saturating your network. On the flip side, I get upset about 0.5% packet loss on my fibre connections - as a point of reference, loosely speaking, less then 1% is OK for VOIP, above
   that not so much. If you expect to be able to use Skype or Viber or what have you on your connection then 5% packet loss is not OK. For just Web browsing it may just about suffice.

4. As an ISP, I want to see the results of an MTR, which show the latencies and packet loss between the destination - this helps me look where the bottleneck might be and is a good first step. I would also like to know when the test was done so I can correlate it with the customers other usage and what is going on on the system. The packet loss graphs you have done are useful as well, but not in isolation.

   As a client, my ISP has been unable to excuse away my graphs, which plot packet loss (I do it for 250 pings, once a second over 5 minute intervals, combined with minimimum, average and maximum latencies for those pings). I also have a set of graphs showing my utilization of the link, and have sets of graphs showing local (ie very close to me), and to another POP they own of specific interest a few hundred KM's away.

Other observations:

It looks like your latency increases in the afternoons - which means the first places I would be looking are if the problem is WIFI when everyone around me is using it. After ruling that out I'd start questioning my ISP about oversubscribing the connection.

This more than likely is a result of congestion somewhere along the line. It could be your router but more likely an upstream provider.

You don't state how you're doing the 50 pings e.g. what time interval, are you waiting for one to fail/succeed before the next or firing 50 off all at once (flood pinging).

Such a loss during periods of high congestion is not unusual in my experience. It can be down to lower prioritisation for ICMP traffic but more likely is happening for the same percentage of all connections - it's just TCP will gracefully resend and reorder packets so you are less likely to notice.

To gain a better picture of the situation I would suggest you implement the following:

1. Increase the interval between your pings
2. Ping an IP address for google not the domain - google.com will return a number of A records and it may be you're using different end IPs (and hence routing differently) without knowing it
3. Record the mean time to respond; see if this correlates with loss - if it does so you see higher ping roundtrip times and higher loss then it indicates congestion. You could then investigate by storing traceroute logs instead and see if there is a likely bottleneck somewhere where you're seeing suddenly increased times
4. Try pinging more than google. When I've benchmarked network performance in the past I've done it using 4 or 5 good end-points (again with the IP address not the hostname) so you can rule out congestion or a specific problem within just google's network causing you to question your entire connection

This is typical of most residential ISP accounts. You're seeing a peak due to network congestion as people get home after work, then go on-line for the whole evening. This sort of evening peak is especially pronounced in high-tech communities with lots of on-line gamers (like where I live, here in Redmond, home of Microsoft.)