When someone catches it from drinking raw milk with the virus in it, it might affect other people too.
What we really do not want to happen is for human-to-human transmission to start, and every time a human gets infected, it helps create a risk for a mutation that starts that going. We know that the virus is in the milk supply right now, so it's not just some distant hypothetical.
If you remember, we had a lot of criticism for China over the role that wet markets with live animals might have played in starting the jump to humans on COVID-19. I'd rather not turn right around and have us be responsible for creating a window for a new pandemic due to doing something similar.