Suspension and lightness. The two things that always make a car faster.
In the grand scheme of things, Subaru suspensions systems are not that great. Out of the box handling is mediocer at best, they are heavy and underpowered. The only thing that really saves a subaru is the AWD and how much more forgiving than a 2wd system it is. Personally, I think the OP is going to be dissapointed when he tries to compair his Civic and an Impreza. The differences will be there when daily driving, but get into a motorsports situation, and they are in completely different leagues. I think that if the OP was moving to a WRX instead of an Impreza, the give and take would be more equal. The handling still wouldn't be on the same level but the seat-pressing fun would definitly make up for it.
But all the arguments get thrown out the window when you need to drive it in the snow. You simply can't compair any 2wd to AWD. Really, in the end, you can't compair them at all. Completely different animals, great for different things and reasons, and people like what they like. You may like an Impreza more than a civic, you may not.