Problem is that asteroids are very hard to see, as they are both cold and dark, meaning they don’t stand out against space very much at all. And even a micrometeoroid poses a risk even when traveling at low velocities (e.g. someone orbiting earth, the meteoroid itself has a relatively high velocity). Getting hit by a 1cm meteoroid at warp 1 would be devastating.