That's why there are 2000 Rust servers to choose from on the Rust Master List unfortunately.
Why do you even care? No one plays on those servers the kids who run them and their friends? It has no impact on other server pops because they don't have a reason to go to Random Kid Sever #1210
There's probably a multitude of factors behind lower numbers on servers.
1. Going by Steam numbers, overall and peak players in Rust have dropped since July/August by substantial number. About 1/3rd (150k down to 100k peak). Those numbers were likely boosted in July because of the OTV server, getting some people into playing Rust for a bit. This mirrors a lot of what was seen at the start of the year, where Rust saw huge amounts of increases to player numbers because of the popularity that the OTV streamers were giving it at the time. This declined back down over the next few months, only to jump up again in July.
2. COVID restrictions are being lessened in a lot of countries and were quite likely a major reason for Rust's more recent popularity in the last two years, March 2020 saw a very decent jump in player numbers on Steam which is about when countries started implementing COVID restrictions.
3. Patches, August patch was pretty interesting, adding subs and underwater content, but September's patch was pretty minor and not a lot of interest got changed. October's camper van is nice, but it's not really an attention grabber and I'd expect most of the patch will be QoL fixes/updates. Great for current players, but it doesn't really get people picking the game up again or buying it.
Taking all of that into account, it's not strange to see smaller servers (50 players or under) dropping numbers. As numbers drop due to people being able to go out and do stuff again or losing interest, players want to find more populated servers to play on, growing larger servers but starving smaller ones.
But yeah, sure, 0-1 player servers run on kids computers are the real problem, right?