Open with Birthmother, not with Birthfather

We have an adopted 15-month old son. We have an open relationship with the birthmother where we have quarterly visits. We do not have an open relationship with the birthfather. It would not be in the best interest of our son to do so. The birthfather has not been cooperative with the adoption and the birthmother has not see him since before the baby was born. We want to share positive things about him (what few we know since he's been difficult), but aren't sure how to handle this. He will know his birthmother, but the birthfather won't be a part of his life. I have a couple of books on open adoption, but haven't found anything that deals with our situation. What's the best way to handle this when our son is older and begins to ask questions?