I'm not a gun person, but it seems to me like anyone who claims they can shoot down a drone with a rifle is also revealing themselves to not be a gun person.
I just watched a YouTube video of ukraine drone operators and they had a guy in the middle of a field shoot down 3 fpv drones. They got him eventually but it is possible...