I have a few inches of rocks and then a few inches of decomposed granite underneath. Below that is dry Southern California soil, which looks like old fill dirt with some buried glass and beer caps.
On the other hand, removing the rocks allows earthworms to access your soil more easily, which is beneficial.
That’s what I was thinking too. A few inches of gravel and decomposed granite will likely discourage earthworms from settling in. Unless you add them from the top and make the soil deep enough with plenty of organic matter for them to munch on.
I also do trench composting in my garden and sometimes bury old wood mulch into the soil. The rocks in my garden have made things a bit challenging.
I’ve done this before, and I prefer my garden beds to be slightly underground to prevent dirt from washing out at the edges. Removing a bit of gravel would help achieve both goals.
I’ve read that a gravel layer can disrupt a healthy soil ecosystem. While it helps drain soil in houseplants, preventing root rot when watering weekly, it can be problematic in garden beds. Gravel causes rapid drainage, which can be disastrous during summer growth periods. I aim to encourage deep root growth and allow rain to seep through naturally, without hitting a gravel ‘dead zone.’ Essentially, you want a natural soil structure in your garden, not an artificial drainage field.
It depends on where the original poster lives. Boreal forest areas up north don’t have native earthworms; nearly all species in Canada are invasive, often spread by fishermen using them as bait. 
In these regions, you won’t see worms unless you introduce them yourself. Therefore, leaving the gravel might be fine and could even be preferred.
It depends on what you want to grow. Plants with deep roots won’t thrive. I had to dig up my garden and sift out the bottom layer of gravel because it caused problems
…or you could just make raised beds about two feet high. That’s what I prefer—less kneeling or bending, more sun exposure, and it creates a nice spot to sit while enjoying your garden’s produce. But hey, I’m just a simple person doing simple things, until I decide to change it up.