Needs
Before you start selecting and investigating the virtually endless possibilities, first determine what your needs are. It really makes a lot of difference and should be the number 1 decider on the parts and systems you pick. A few typical systems are: gaming PCs, video editing PCs, Media PCs and all-rounders. To help you with this I will now talk about each part of the computer and tell you whether you could need a bit of a boost in that particular area.
CPU
This is the processor of the computer. The CPU takes care of loading, helps most design packages and is vital for a speedy file conversion or editing capabilities. Currently the best available is the Intel i7, but on a tight budget AMD alternatives could come into play. Focus on buying a good processer - minimal quad core 2ghz+ ghz - when you want a PC that makes graphics or edit videos. Gamers should also invest a little extra in this area.
Memory
The quality of the RAM (memory) is determined by four factors: type, amount, speed and timings. As of writing the rule is to always buy memory of the type DDR3, of 4gb or more, with a speed of at least 1066mhz and timings that are faster than 9-9-9. Remember: more memory is better, faster memory is better and the timings should be as low as possible. For video editing systems try to get 8 gigs of ram that meets or exceeds the above-mentioned criteria. Game PCs and all-round PCs will work fine with 4 or 6gb ram respectively.
Video Card
That's the big one located in the middle of the case. There're two brands worth mentioning: Nvidia and AMD. The world of video chips is a difficult one to understand, mostly because the branding and pricing say nothing about the actual quality of the card. Weird huh! Read benchmarks of the various contenders and take into account the performance, price, cooling and energy usage.
The GPU is the most important aspect of a gaming PC. Buy a great one (or two). Amd currently has the edge. For video editing PCs and all-round systems middle-range cards will do, so cards that are in the $80-150 bracket.
Other parts
Of course a computer consists of a lot more parts but these are generally more straightforward. For the power supply do some research and get what you need plus a little extra. Don't go overboard! Or you'll lose efficiency that will result in an unwanted high-energy bill.
The PC case has two aspects you should consider. Try to get one you like to look at, and one that will cool your system without going to Apache decibel levels.
Finally buy a good motherboard that will hold your components. Take your time doing some research on this because it's really important. Other than compatibility few factors come into play. Some boards allow for better future upgrades though.
Budget vs needs
It's a constant battle for PC buyers, what is the budget and what does it need to do. You should stay realistic and don't buy above your weight just to get that one cool part. A great computer is always in balance, with emphasis on curtain parts depending on your needs.
And those are the aspects you should consider when buying a new computer. So to recap, make sure the components match your needs, and always try to stay true to your budget. It's very easy to go overboard on this one, trust me. Good Luck!