adoption hurts child?

I guess I want to know if I am doing the right thing? Here is my story: I was living on my own (with my 2 children, boys) after my divorce of a 10 year marriage. I met a man and things went well. Very well and we eventually got married. He has 3 kids. 2 from a first marriage and 1 from a second marriage. The one from the second marriage had a mom who was financially stable and shared custody with him. They had joint custody and saw their son 50/50. He paid little childsupport but he also had the child half the time. He wasnt asked to pay much for that reason. Anyway, out of the blue she asks if we are happy together (after we got married) he said yes and she then ask if I would adopt her son. She said there was too much annimosity and people were turning her son against her (i did NOT see this and I always was supportive of showing her in a good light when he talked about her on our visitation days) Anyway it was a shock. Come to find out she got rid of her other two children from previous relationships as well. I agreed. I adopted her son with only one incident of her saying she was driving to get her son (i believe she was drunk or on drugs) the next day she oppologized said she still wanted to go through with it but offered no explanation of why she did it. She ask for certain days alone with her son leading up to the finalization of the adoption to say goodbye. She called us on the first visitation day and said she wanted no more as there is no reason to prolong it. we told him mommy was sick and couldnt be a mommy anymore so I was going to do my best to be his mommy. I thought that would releave her of being the mean person as she is "sick" I could later explain it as a mental illness. She left a box of stuff for him, seemingly "special items" although there was nothing special written on any of it. After awhile she moved and has had nothing to do with him. I then started to have trouble with his father. who he was came shining through. ALL at once. I found out he was cheating and lying and so I left and was divorced in a few weeks. At that point he started saying he would never see me again, then he was just angry and wanted me to be his mom. It was always depended on his attempts to get me back. I was so upset I got a rash from stress. Everyone started to say LET THE CHILD GO. Its not helping him or you to go through this. Well my ex now says, he wants to change. Its been almost a year. We have both dated other people and I am willing to work on the relationship. Now we are going through problems again. We have only been together for a few months so there will be isssues as something that has been hurt like he hurt us is hard to get over. We are going to counseling. But again he still lets me know I will not have my son if we split. IMO i adopted him and its my responsibility to fight til I die for him. My Doctor who treated me for hives, and my family say MOVE ON from both of them. There are other issue such as my ex's family is nosey, possesive and controlling and have made me the bad guy and tell the 5 yr old that they dont like me and that I ABANDONED him when I left. Whats your opinion? Right now he is at my house all week and my ex comes on the weekends. Im loving being mom to him, even though I have to work through all the behavior issues created when I was gone.