I've done it a couple of times - 2007 & 8. It's a brilliant day out in stunning scenery and definitely a challenge worth setting yourself. There is no reason you can't achieve it. The relief and exhilaration of reaching the end at the top of Alp d'Huez is enormous. You'll have to get some long miles in and some long climbs. There's plenty of advice around on training for sportives such as this - I was doing 200+ miles a week leading up to the event including one long ~100 mile hilly ride at the weekend.
Ditto Arthur and AndrewG. I tackled it in 2005, about 18 months after taking up cycling again (after a very long break of almost 40 years). All I can suggest is get in lots of miles, lots of hills and, if at all possible some experience of alpine climbs. If I can get from, more or less, scratch to the Alpe D'huez via glandon, telegraph and galibier so can you - and you'll have a fantastic day.