Which plugin format should I use?

It depends on the Digital Audio Workstation (DAW) you own.

Here is a list of the most famous DAW compatible with each format :

• VST (PC/Mac): 
  Cubase, Reaper, Ableton Live, Acid Pro, FL Studio, Sonar, Studio One, Nuendo, Samplitude
  
  
• AU (Mac only):
  Logic Pro, Garage Band, Reaper, Ableton Live, Studio One, Digital Performer
  
  
• RTAS (PC/Mac)
  ProTools (for version <= 10)
  
  
• AAX (PC/Mac)
  ProTools (for version >= 10)