So Upset and Need Support

Long story short (as short as possible anyway) My son will be ten this year. I was married to his father who decided to cheat while I was four months pregnant, which ended up in a divorce while I was still pregnant. Ex Husband saw my son once or twice as a very young infant, and then again once time at my son's age of three. Since then (7 years) my son's father has never asked to see him. in 2005 I contacted child support services to establish a support request, which my ex then called and BEGGED to stop the case, and that he would send me money outside of the court system, which he has. Six, close to seven years later, he has paid money monthly, but NEVER spoken to my son on the phone or asked about him etc. no concern, no calls for birthdays, holidays etc. While all of this is going on, I met my now husband in 2004 and we just got married in march 2011. We have discussed adoption by my husband of my son and my son and husband are both thrilled. (as am I!) On top of this my son has special needs. If anything ever happens to me, my largest fear is that the only person who has legal rights would be my ex-husband, who has no clue how to take care of my son. My now husband and my son's "true" father, knows everything about caring for my son and has for the last 6-7 years total. My concern is this, I had some terrible lady tell me that if my ex-husband does not sign over rights to my son, that the adoption will not be allowed. Is that true? After ten years of not asking to talk to, see or be a part of my son, who does have special needs, would a judge truly not give my Husband and Son's stepdad paternal rights? So upset, scared and worried. I feel like someone ripped my heart out when she said that to me.