Birth Mom Visit

I'm new here and hoping for some advice. I didn't really see my exact issue discussed already so I decided to ask my own question. I apologize in advance if this has already been discussed here and I'm also sorry it's a long post. Anyway, I really need advice on how to handle planning a visit with our daughter's birth mom. We were at the hospital when our daughter was born and she was placed with us immediately. Her birth mom lived about two hours away from us for the first year and a half and so we had some good visits with her birth mom in that time. When our daughter was about a year and a half my husband's job changed and we relocated (11 hours away from her birth mom). Because of this job change our financial situation changed quite significantly as well. We kept telling our daughter's birth mom that we couldn't afford to come back for a visit and that when we could afford it we would let them know. They are really mad that we moved so far away, but we truly had no option. Last fall, when our daughter was three, we went back to our previous city and tried to get in touch with her birth mom for a visit. We couldn't get in touch with her and found out later she moved out of that state. We also, went back earlier this year and again tried to schedule a visit but she was still out of the state. Now she has moved back to the same state we all lived in before and our daughter's birth mom's mother contacted us about meeting half way so neither of us had to drive the full 11 hours. This sounded good to us and I told her I'd get back to her with some dates. Well, my husband's job is more demanding during the summer and so between his work responsibilities and some time that her birth family is unavailable I had a hard time finding two dates together this summer to meet half way. We would have to drive half way for the visit, stay in a hotel and drive back the next day because it's too much all in one day even going half way (this would also be a strain financially). So, I told them that we are planning to return to our former city for a wedding this fall and wanted to meet with them then. I figured it would be easier on them because they wouldn't have to travel as far and we are going to be there for a few days. Well, they didn't see it that way at all. Our daughter's birth mom is pregnant again and due a few weeks after we will be there. She said she can't travel two hours that close to her due date and said she's upset because it's been so long since she has seen her. Her mother also contacted me and basically insisted that we had to find time to meet them this summer. I'm really at a loss. I want our daughter to have a relationship with her birth family, but it seems like they don't want to work with us and/or they don't understand where we are coming from. I also really hate it when people are mad at me and I just don't know how to make it better. Have any of you experienced something like this? How have/would you handle this situation? I really don't know what to do or how to respond to them.