It definitely helps to have a good athletic background, but Crossfit does an amazing job of welcoming beginners. Honestly, I can't think of any other community that does such a good job of simultaneously pushing the most advanced to their limits while still welcoming beginners.

Most Crossfit affiliates have an on-ramp program that will get you, no matter where you're starting, to the point where you can participate in full workouts.