After two years, still not attached to my child

We met our sons' "babymommy" as they like to call her, when she was 5 months pregnant with our younger, "E". She also had a two-year-old, "J"; the boys have different fathers. We took E home from the hospital, and a few months later, she asked us to take J as well, which we did. We "bonded" to J in the time we were getting to know them (we saw them weekly, either doing activities on the weekend or when I'd take her to her ob/gyn appts), and so when she asked us to take him, there was no question about it. We would do it. Later, I could look back and see when the shift between J and I actually happened. It happened when E was born, and when their mother couldn't take care of J while she was recovering from the c-section. We took J home with us for the first two weeks of E's life. We showered both with attention -- everyone who came over did, because they were also excited to meet J, but J had decided at that point he wanted nothing to do with me. It's almost literally two years later, now that we're approaching E's second birthday, and I'm having a heck of a time attaching to J and vice versa. It has improved for sure, but he's not interested in me and I'm not interested in him. We move around each other like boarders in the same house, but we're not family. He brings out the worst in me, and no one since my Very Bad Ex-Boyfriend over 10 years ago manages to do that. I find him unpleasant and unfunny and irritating. I know all the reasonable and logical things; and that as an adult it's my job to overcome those barriers. He needs a mother--he needs ME. But it's too hard, and I just don't know what to do about it. We've been going to an adoption therapist for over a year now. Therapy is helping, yeah, but only as a weekly reminder not to dismiss him completely. Not to be impatient. Not to be mean. Not to be negligent. But the times I see any charm or real emotion in him are few and far between. It doesn't help matters that he's perfectly lovely to DH, and that E, whom I bonded to instantly, loves and needs me so much. Our therapist did explain that he's testing me. That he lost a mother, and he assumes he'll lose me, so why bother attaching to me. Instead, he tests, he pushes, he neglects, to see what I'll do in response, so he can prove himself right and protect himself from another loss. I know this in my head, but remembering it 24 hours a day is too hard when he's being a brat, being rude, being petulant, and being attentive and affectionate to DH while giving me the stinkeye. Our therapist showed us a video on The Wounded Child, and it resonated deeply, and everything applied to me. How there was no connection between him and me; all the dreams of having a new child dashed in that first real exchange between adoptive mother and adopted child; knowing that you will never really know that child; and to a certain degree, fearing having that stranger growing up in your house. Knowing that I'm not the only one who is experiencing this, that it's common, that it's more than common, doesn't make me feel better. It also doens't help that I feel alone in this battle. DH does nothing to support any development of a relationship between us. He doesn't ask J to interact with me more or any differently, whereas I'm often encouraging E to hang out with his daddy since he's a momma's boy right now. DH is happy to be J's hero and do everything J asks him to. He doesnt' interfere with my parenting or disciplining, but I can feel it on the back of my neck that he disapproves, doesn't agree, thinks I'm too strict, and yet I think he's too permissive. I've seen in other posts that the rest of you were able to attach after several months, but seriously, how long is this going to take? It's not for lack of trying!