Why aren't they using drones and more automation?

The in-universe answer re: drones would be that people want to explore. Sure, it's dangerous, but it's also exciting, fascinating, and fulfilling. That said, I feel like a responsible captain would make much more extensive use of probes than any of the shows.
Re: data streams, I don't have a good in-universe explanation. I have a similar question of why they don't have security cameras in all the hallways and public areas.
Also, using the transporter to go down to a planet always runs the risk of some storm or an orbital threat stranding your party. Why not use the shuttle as SOP? It gives your away team more resources, both for their mission and for an emergency.
- The cameras are more of a privacy issue that I imagine the Federation tries to uphold.
  
  There was an episode of TNG where a "passenger" got onto ship's comms and was contacting Picard on the bridge. When Picard told the guy that the comms were reserved for ship's business, the guy asked why they weren't restricted, if that was the case. Picard said that was unnecessary as people in Star Fleet generally just...behave themselves.
  
  It's a military / government ship. There is no real privacy.
  Everyone can read your personal logs if there's a good enough reason. Anyone can just ask the computer where anyone is at any time. People can just barge into your holodeck program. Anyone, from civilians to bartenders can just call up the bridge and talk to the captain whenever they want. People are just expected to control themselves.
  I think of it like how people don't need to carry defensive weapons now, while a knife was very common in the past. People are just expected to control themselves and not rob random strangers today.

I can't remember who said this in the show in Universe; maybe Janeway? But I think a similar question was posed, and the answer was that nobody would have anything to do if exploration was entirely automated. It's fun and exciting and gives people's lives meaning.
My headcanon is that many mundane things are automated, and we don't see them because they aren't plot-relevant.
- They are mundane and automated until something goes wrong, then we get a holodeck episode.
- Like pooping
  
  I hate that this was the first thing I thought of. :)

I've always said that Starfleet is, first and foremost, a jobs program.
It gives purpose to people who can't find their own, in a time where your needs are provided-for by default, and seeking personal fulfillment is the purpose for most people's lives.
Drones would cut out the human driving a shuttle over to inspect an anomaly or object themselves, robbing them of a sense of accomplishment and achievement. Starfleet is about that stuff, so that's a no-go unless nobody wants to do it and it needs to be done anyway. We see that a lot, too. They do have probes and sensor stations and stuff, after all, usually in really boring and unfulfilling locations.
They have excessive, ridiculous redundancy. They have people doing jobs the ship computers could (and often, in times of need, DOES) perform very well on its own. There are several recorded instances of entire starships being successfully maintained for extended periods of time by a single individual (who does go insane due to isolation every time, because plot).

1 argument against the use of drones:
- See also: any time an AI has been given command of a vessel (except Data, and even then he caused problems a couple times).
  
  "That was the stun setting. This is not. I can reduce this pumping station to a pile of debris."
  —Data, the most sophisticated android in the federation, conducting diplomacy
- Before that, M5 was one of the worst disasters in Starfleet history.
  
  And although it was probably at least partially covered up control wasn't exactly great.
- Please explain, this is a starfleet drone attacking a starfleet ship? Was the drone highjacked or just went rogue on it's own?
  
  Option B
  Rogue AI hidden in code.
  
  Thats drone ships going bad from lower decks
- And more importantly, the drone MISSED data that revealed the planet was inhabited!
  Not Star Trek, but the Three Laws of Robotics is the textbook on why AI and any strict programmatic interpretation of Rules is a flawed goal.

No one's going to watch a realistic exploration sci-fi show about small unmanned ships quietly going about their jobs with no drama.
- You sure about that...

They tried to, but the exocomms became sentient and they couldn't be used as slave labour anymore.

You forgot to add "Are they stupid?" at the end of the post title
- 😂

Why aren't they using drones
They use probes all the time... Those are basically drones. But better.
Also: The Borg have several drones. 😏
- I mean i just watch a the next gen episode where some science guy had created a rift in spacetime and instead of sending in a drone/probe they almost got Lieutenant Data killed. Another thing I was wondering why aren't they backing up Lieutenant Data?
  
  Having also recently seen that episode, they send Data in because he's the only one who wasn't confused by the time weirdness. Picard even tells him sending others would only slow Data down and if they should get hurt it would make the time sensitive mission even more difficult.
  As for why they don't have more Datas: They don't even know how he works. The dude who made him and Lore died without sharing his research.
  
  One thing to remember, is that the concept of drones and AI, how we are currently developing and improving those, was not something people back then had on the radar.

In a similar vein, I've been rewatching TNG and find myself thinking that they really should have put a cctv camera in engineering. Could have saved them a lot of trouble.

Probably because the butlerian jihad forbade them.
- This may be in jest, but they actually fully made this an actual plot point in Picard.
  
  I gave up on Picard. :(
- butlerian jihad
  dune is a trip! 😂

Why invest in tech when there are so many red shirts around?
- 😂

They aren’t?
Enterprise sends out probes (drones) just about every other episode, especially in TNG. Almost everything is automated on the ship, controlled through the computer interface.

Drones can't negotiate the if they trespass or make accidental contact in uncharted territory.
- I mean, given the current trajectory of software and hardware, they probably could.

In the original series episode I just watched, they reference that they've sent out tons of unmanned drones/probes to map out systems and planets, but starships are enormous and better equipped, so they follow up on any readings from the probes that seem interesting. If there's an in-universe answer that isn't "it makes better television", I'd say it's a combination of:
Space is really, really big, so probes are only covering small areas anyway.
Their mission is to explore and contact new life, which is more likely to be successful with a human touch.
- Space is really, really big, so probes are only covering small areas anyway.
  That's backwards. Probes can always cover a vastly larger area than manned ships, so needing to cover more area is always a reason to invest in more probes rather than dumping resources into a handful of very expensive ships.
- Their mission is to explore and contact new life, which is more likely to be successful with a human touch.
  Have you met us? ;)
- Also, a lot of the niches served by drones are already covered by AI driven holograms. If you're not worried about bulky hardware and can supply holo-emitters on the cheap via replicators why wouldn't you?
- which is more likely to be successful with a human touch.
  Andorians and Tellarites: "Are we a fucking joke to you!?"
  Vulcans: "We must expend great effort to suppress our feelings of amusement at all of you."
  *Federation civil war begins

The Amazon drone wars of 2032 is still a touchy subject in the future

It's fun to see what modern tech is missing from decades old Sci-fi.
One of the most interesting ones for me, is that there aren't any screens with text on them shown in the original star trek. That's because when TOS was made, computers communicated by teletype/printout. TOS is older than the concept of text on a screen.
That said, I feel like a drone would be part of a tricorder. I have a DJI drone with a good camera, and I use it a lot for getting pictures of things that are out of my reach, if you had one paired with a tricorder, you could look at things out of your reach.

I know this is how people in the 80s and 90 imagined the future and a lot of concepts were probably too far fetched for them.
You also have to consider that TV executives were also considering this, and punting any ideas they thought wouldn't be accepted by a TV watching audience of the 80's/90's. Like the planned gay characters who were scrapped.
I mean, think about that, being gay in the future was too much for some television executives to accept, I really wouldn't be shocked if they gave thumbs down on lots of more esoteric and abstract episode concepts simply because they thought it would be too above the heads of a 90's TV audience.
And to be fair, they were probably right. The communicator seems less amazing now that we live in a world with cell phones, but back then a personal communications device that was on your person at all times seemed definitely in the realm of sci-fi. Now we all have a near-equivalent in our pockets, as well as it being general purpose computing device that can be used as a personal communicator and much more. Our communicator is also a primitive tricorder.
Some of the ideas they did let pass were either already accepted tech from the original series or were close to existing civilian or military hardware that was in it's infancy.
So a combination of "this was the extent of human imagination about these concepts back then" combined with "television executives are keenly aware of ideas the general public won't understand, and doesn't like confusing audiences, and thus will cut any content they deem too abstract or confusing" is what I think actually happened. One part actual limitation of imagination, one part purposeful limitation of imagination as to not to confuse the audience.
Which, honestly, is fair. Do you think sci-fi series like Rick & Morty would exist as they do without all previous sci-fi series laying down frameworks we understand for it to be based on? Human knowledge and ideas do build on themselves, and so, in a way, the TV executives are half-right that you can't overexpose an unexposed audience. You kind of have to slowly spoon feed them ideas over time.
Like, what if we tried sending Rick & Morty as a show back to the 1960's, and how many of the ideas would be entirely over the audiences heads? Simply because they didn't have 60 years of sci-fi media relating different iterations of these various ideas until "the multiverse" is just talked about like it is just a given thing that exists, and nobody questions it. At least a few would have trouble wrapping their minds around it, because while many of these ideas were pioneered in the Original Series, their lack of depth might leave audiences back then really confused about some of the ideas presented.
- To further add to the idea that the concepts themselves were not foreign to people at the time, just read some classic scifi from Isaac Asimov or Ray Bradbury. Or even older than that, and check out some of Da Vinci's ideas. There are even ancient Greek writings clearly describing the idea of many modern inventions we take for granted today. People are rather imaginative and inventive, and can generally take a simple idea to extreme heights long before we have all the necessary knowledge and tooling to make it reality. Even now, we know how we might do a lot of stuff only seen in fiction like warp travel and Dyson spheres, nanotechnology, etc. We just haven't got some of the requirements to actually do those things nailed down yet.
  
  Too right, exposure to those kind of ideas has grown over time, and thus given the modern era the ability to take those ideas mainstream, because of the simple breadth of media available. We often take it for granted that even a hundred years ago, it wasn't super easy to get a hold of books, let alone catch every film release. Now a near infinite stream of media is literally available at people's fingertips. The speed and amount of media that exist has contributed heavily to a more informed modern audience that can digest these ideas more easily, because they've simply been exposed to more media explaining the basics underlying such ideas.
- No human being alive in the 1960s could have survived the amount of drugs they would have needed to ingest to create Rick and Morty in the 1960s.
  I feel like if you built a time machine and took Rick and Morty back to the 1960s, it would have just looked like flashing images on a screen or a nightmare straight out of hell to them, their minds would have not been able to process what was going on not because there's any real depth to the series but just because we have so much exposure to the topic content that we are able to process it where is the closest person in the 1960s would have had is a few episodes of the black and white Lost in space or a little bit of Twilight zone maybe.
  
  Exactly, media moves so much faster now, so they literally had a smaller frame of reference and were exposed to far fewer of these ideas than modern audiences. We take it for granted now, but it used to be difficult to get your hands on media that was more obtuse or complicated, because often they didn't have copies at your local library, and as such, audiences back then just wouldn't have the frames of reference that we do in allowing us to understand the concepts and references to other existing media.
  
  Ask me how I know you're not into foreign and independent film of that time period, nor from the several decades preceding it.

I mean, the Borg are right there on the poster!
They live in a post-scarcity world with insane science fiction technology, but they keep the executive decisions in the hands of crew members.

What's up with Troi's nose?
- This image smells like AI art a bit, dunno maybe I am wrong.
  
  You're right. Troi's and Data's hands are messed up, Data has unreal wrinkles on his forehead, the shadow on Picard's neck seems to be a dent, and of course, Troi's nose has a different camera angle on either side.
- Cocaine?

The same reason all the top ranking officers go on all the away missions; so the show isn’t boring.
- They also don't have armies in star trek, like, if it makes senses to send 5 armed people why wouldn't it make sense to send platoons or batallions in some cases? it is always individual enemies they never seem to face entire armies on foot.
  
  So, this one is a little more justifiable.
  Phasers, and starships. And because ST isn't about war; it's about diplomacy.
  With the exception of the Borg, there are no personal shields. One person with a phaser is a WMD; it's a sweep-able, incredibly destructive weapon, and if you're not interested in merely maximum casualties, it also has an effecting area-effect stun mode.
  But the other reason is starships. Ignoring plot devices, orbital targetting is incredibly accurate and starship phaser banks can anihilate entire areas. If you tried to move an army across an area, it could be vaporized by a single enemy starship.
  Either the enemy has superior technology, in case it doesn't matter if you send 1 or 1,000 people; or they have inferior technology in which case you need only one person. And if you have air (space) superiority, you can level cities with even a smaller starship.
  So ST is more like special forces making surgical strikes, sending in problem solvers and diplomats, and hoping nobody starts shooting in earnest.
  Because phasers are OP.

PIC touches a bit on this subject, spoilers ahead:
The first season has artificial intelligence and the attempts to (re)create a true artificial life form like Data as one of its main themes. Also an explanation why it isn’t and hasn’t really happened so far, anywhere in the quadrant(s).
There is also the stated reason that simply cataloging space with probes or so is not enough, mankind (and in extension, starfleet) consider in-person exploration much more rewarding and fulfilling, and a worthy occupation in itself.
As for data streams from their away teams, yeah that would certainly make sense. Even today's military is essentially fully online. Camera and audio feeds, gps access, depending on the soldier’s function full drone control suites, various other situational devices.

I know this is how people in the 80s and 90 imagined the future
Hum... It's an adaptation of a dumbed-down adaptation from how people in the 60s imagined the future.
The 80's scifi was mostly dominated by cyberpunk, that is completely different.
- I call TNG "Adult Contemporary Futurism"
  
  That describes the TOS better. TNG was clearly retrograde futurism when it was launched.
  That said, TOS was also a bit retrograde, because people were crazy about robots, huge computers, and virtualized worlds by the 60s.

You probably hit it on the head about the limits of people's imagination in the 80s/90s, but also I wonder if it had anything to do with Roddenberry and his opinions of what was / wasn't going to show up in Star Trek? He famously forbid interpersonal conflict between federation people, making it annoying AF for the writers on TNG. Plus drone ships exploring everything might not make for captivating TV
- You are right, it would remove all the stakes, but then I feel that is just bad/lazy writing, If you really went into such a future with your imagination, I am pretty sure it might even raise the stakes as there might be probes hiding/sleeping in places you do not expect or in general all of that while making exploring a planet more secure, it could then make for a great twist if all of the sudden a swarm of drones dropped out of hyperspace right on top of them.
  
  I seem to remember at least one episode about a rogue drone missile (nuke?) of some type that they had to board / disarm, but yeah that aspect of things was really ignored. Same thing with space infrastructure / communication relays and all that, never hear about it unless its a hook for a specific episode..