Yes, but on the flipside, speccing in Wrath a lot will give you faster digging and claiming Imps. I'm guessing it'll still be very much worthwhile to at least pick up Summon Worker, but I think not going into Greed too early can easily be worth it.
What do you mean with middle vein skills and Greed's juicy set? I don't think it's particularly more attractive than the stuff in Wrath or Sloth - only in the long run, but it should work that way