Did you measure this ping? If so, that may not be accurate. ICMP processing is a low-priority task. The actual routing is done in silicon. You could ping a ping of 1000ms but a packet forwarding latency of 500 microseconds.
One of the things that can consume a lot of CPU time (but not affect packet forwarding) are spanning tree issues and processing lots of multicast joins.
Next time grab a packet capture for a post-incident review.