I wouldn't be concerned about him if your doctor isn't. Ultrasounds aren't a exact science, they gave you a estimate and it isn't always right. Try not to worry...a small baby doesn't mean he will be a small man. Best bet...talk to your doctor.
Best of luck.