Step Parent Adoption With a Restraining Order

Concerning a Step Parent Adoption in California: I'll try and keep the background info short and sweet so I can get to my question. 7 year old daughter - she is autisticEx husband only took part in raising her from about 6 months of age - started falling off caretaking shortly thereafter due to meth use. Beat me up twice (ex wife) on top of beating up a total of 4 other women in total and serving multiples terms in prison/jail for DV. Latest charge in 2015 against current wife. From 2014-2016 attempted contact with me for child twice a year. Each time appeared that he was more interested in contacting me somehow because he was homeless, or wanted to see what I was doing. Never sent gifts, never sent money. Last attempt at contact 2016 - wanted me to come to his mothers house to pick up VHS videos for child but long story short - had to consult with my attorney and the police it appeared to be a set up to get me to violate the restraining order I had to get ( it's a 5 year order). He has not made any other attempts since 2016.Before he was arrested on the last DV charge - he showed to the divorce hearing, but did not show up to mediation for our daughter so I have sole legal and physical visitation is at my discretion. My question is my current husband and I are planning on filing in July for adoption. I know in California it's one year no contact. We have definitely got that. My fear is he will contest but he will say it's because of the restraining order. I know legally I have no right to deny him visitation, in fact I never did, but with the domestic violence being so severe I couldn't put myself or my child in an unsafe situation, and every time I got the two contacts a year it always came down to something to do with me. I've talked to a few attorneys and they said with the lack of contact, my daughter basically being raised by my current husband since she was 2 and knowing him as "dad", we should be fine if he tried to contest. He doesn't have much of an excuse. If I was "denying" him visitation, or if he felt uncomfortable about contact with a restraining order , he could easily go to court and file a modification. He has used the self help counter at the courthouse to get aide when he tried to contest our divorce, and his current wife paid over $10,000 for a lawyer to get him off his domestic violence charge against her. So he had resources to fight for his kid. I was just wondering thoughts on this from you guys. Also, has anyone had a similar experience with adoption and restraining orders? Thanks a bunch!