Need help handling life post-placement

My son is 10 months old, we brought him home at 1 day old and he is absolutely amazing and perfect in every way. My husband and I are struggling so much with our relationship with his birth family, however. We were actually approached by a co-worker of mine who knew we wanted to adopt and whose daughter was pregnant. Sounds perfect right? We thought it was until the birth father became involved and her pregnancy became a back-and-forth rollercoaster from hell. It changed weekly, sometimes daily and every time we got a text from her our world would turn upside down. We met with the birthfather and his family and they said to our face that they would fight the adoption with everything they had, no amount of money or time would stop them. When he was born, I was in the room but the experience was horrible. The entire hospital experience was horrible. The agency was sure we wouldn't take him home based on their experiences with the birthmom so I didn't know why I was there, it was torture. She changed her mind constantly, why wouldn't she in the hospital? We did bring him home but then we just waited for the lawsuits and to lose him. We found out birthmom drank and did drugs while she was pregnant, which she lied about numerous times. Miraculously, lawsuits never happened (despite many more threats), baby boy is fine (after some brutal withdrawal we didn't understand at the time) and we are now in the last stages of finalization. We thank God for this everyday but it all has a lasting effect. After he was born, his birthfamily has had absolutely no boundaries. They have come by our house unannounced, they found out about our baby shower and attended without warning, they invite people we don't even know to our house, they text constantly, birthmom wanted to do a photoshoot with our son on mother's day, etc, etc, etc. We went with an agency to make sure we did everything properly and they advise us never to rock the boat or say no to things because our son is still "removable" and his adoption is so risky. Birthmom refused any counselling or mediation after placement even though we'd already paid for it. To add to the stress of the situation, I had to have emergency brain surgery when my son turned 5 months to remove a large brain tumor. I almost died and I'm still recovering. My husband has come face to face with losing both his wife and his son this year. My first thought when they found my tumor was relief that I might get some space from the birthfamily for a while. No such luck... We feel like we have no control over any aspect of our life. We are ridiculously over-sensitve now and feel like ticking time bombs. I used to feel fantastic about open adoption. Now we know we'll never want to adopt again and it's heartbreaking. I want SO BADLY to feel positively about this adoption for my son's sake and to have a positive relationship with his birthfamily. I have to go back to work next month and I'll see his birth grandma everyday, who is angry with me because I finally drew a very minor boundary (told her she couldn't invite people to our house without asking us). I have to find a way to heal from all the fear, pain and guilt that is wrapped up in this situation for the sake of our family. I'm feeling so much guilt and grief because I had pictured and started a close relationship with his birthmom before everything went haywire and I'm so disappointed. Any words of advice, perspective, would be much appreciated, just be gentle on me. Sorry about the novel. I know some people have it so much worse and I would do it all again for the sake of my child, I just don't know how to move past this and make it work.