Pitcher's ERA question?

When a pitcher allows a base runner due to it's own fielding error, why doesn't it count against the pitcher's ERA, if the base runner eventually scores? The rule is that the pitcher shouldn't be penalized if a runner scores due to a fielding error or a catcher's ped ball; but I think it should count against the pitcher in the case when the error is the pitcher's own fault. Does anyone else agree?