Issues with birth family visits

I have some confusion about the visits my son has with his biological grandmother. He is 3 and a half with a diagnosis of autism. He has made great progress since joining our family and there has even been talks of a change in his diagnosis. He was placed into foster care at birth. He is number 5 of 6 children. The grandparents have children numbers 1,2 and 4. Child 5 was adopted by another couple and number 6 is still with the birth mother. Both of the biological parents have decided not to be present in anyway in his life. Im not too sure about how it is with his brother. The issue at hand is that the birth family and other adoptive parents dont seem to be that interested in contact with him. The grandparents have fairly consistently cancelled visits with him when his adopted brother wasnt present. Since my son was still technically in the system he had visits twice a month while his brother only had his once a month and it was at a supervised visitation center since my son was still in foster care. Since the adoption has Our last month of having the visitation center was in July even though we finalized earlier in the month. The other adoptive family went on vacation that month so the grandmother said theres no point in us driving there since the other brother wasnt going to be there. That seemed a little cold to us but we felt it to be important to continue the visits. So after that we had to decide on our own place for visits. We made a few suggestions about visits that no one else seemed to fond of so it was left unsettled. Then the day before the next visit we recieved an e-mail in which the other adoptive parents and the grandmother had decided when and where it would be without even discussing it us. And we did have some concerns because there had been some issues with what the grandmother had done during the supervised visits and the county social worker told us to be careful and not to trust her and even off the record advised us not to continue visitations with anyone during the adoptive placement. So we went ahead with the visit despite the extremely short notice, not being a part of the decision and the fact the location was changed to something more convenient for the family and adoptive parents but further for us(we already drived two hours). During the visits hes not really interacted with just a greeting and goodbye. There have also been issues with people bringing food and gifts for everyone but him. Even his foster mother, who had him from the time he came home from the hospital to the time he came to live with us said that they never interacted with him he would just sit in a corner by himself. I dont want to sever what relationship he has with them but at the same time I dont want him to have to experience this small gradual rejection. Anyone have any knowledge or experience in a situation like this?