When you begin a game, the first things imps do is reinforcing the walls, and the only way to make them stop is to slap them or grab them. This is a bit tricky when you're giving out directions and suddenly begin to think for a second where to dig next, the imps will begin to reinforce the walls. This is kind of weird when I see they can run throughout all the dungeon if I ask them to dig blocks from different places at once, but they won't do it if I click one by one and go reinforcing walls instead. The problem is that reinforcing takes quite a bunch of time.
Although, maybe all this is intended and slapping them is what was planned ?