I finally after 6 months found out that this plugin is the reason some players are capable of placing ladders while in a building blocked zone.

For some reason this plugin allows them to place ladders while in raid block.

Is there any way of fixing thit?

I stand corrected: this wasn't a glitch but it's the way the config works. Been struggling with this for ages. So if you use Building Blocker, make sure to remove the ladder in this config so it works as intented.