i need some advise

Himy husband was adopted and his brith mother found him. We have a ten month old son together and a new one one the way.. Things were going good with both adopted and birth parents until we decided to get married. They(birth)did not want to come because the adopted family would be there. We understood that but the rest of his birth family came and there was no problems but now my husband and his birth mother are fighting and he wants nothing to do with them anymore and has told them that but they still want to see our son. They have told us that they will take us to court to get legal right to visist with him.. I was wondering if they have any legal rights to him??? Very confused