husband to adopt my son as step-parent

I have an issue and wondered what kind of difficulty I might have. My husband and I have been married almost 5 years now and we are finally ready and at a place where my husband would like to adopt my son. He has wanted to do so previously but I wasn't sure if we should yet at the time, since he is new to having a child and of course that takes some transition and understanding of all the care a child really requires. I think it will be a good time now to start the process to give my son the stability he will need as time goes along. I want also to know that my son will be provided for beneficially as my husband has done for some time now. However my son's father has always lived apart and has only tried to see his son once when he was an infant. He has made no attempt other than that to visit him or know him, hence my son does not know him. He has also expressed that he would be willing to consent to an adoption process and said that in 2004 should my husband want to do so. So, I found that interesting that I received that letter at the time from my son's birth father stating that. I do want to ensure that the adoption can go through now. My concern and question, is now, since it's been 4 years since I received that letter from my son's birth father in 2004, would there be any likely hinderance from the process on his part? I don't want him to hold it up but I do know it comes down to his consent or not. My son's birth father does pay me child support and has since he was an infant however and so, perhaps that would be an incentive for him to consent to it. Not to make it sound like he is so callous that would be the only reason. I think he mostly likely may feel bad that he has not made adequate effort to at least know his son. And so, he might feel this is the best option for him. I truly feel that it is. My son, is doing so well in school, is well adjusted, kind, considerate, and has a life, friends, and stability and love of many who know him. I know this will be the best thing for my son, since I make it my aim always to do my best for him, and teach him well, the kind of capable thinking, good and decent person I want him to be. Also: I have always made it plain that I am open for his father to be whatever part of our son's life that he would want to be but he has not chosen to do so. I know now that it is time. My husband takes good care of us and loves us dearly. I am lucky. I hope only to know if this will work out for me. What do you think?