Why we just cant be a family?

My boyfriend and I been together for almost 5 yrs. I have a child with him along with my other children. He has other kids with another woman. Twins eight yrs old. We dont live together, but we do spend time together on our off days when we are not working. Any way his kids mother has total control over him. She tells him the kids cant be around me thats ok with him. She pops up anytime of day with the kids thats ok with him. I feel hurt because he dont think its anything wrong with this. He acts like he is scrared to tell her anything. This Friday we were together intimately and his kids just burst into the room. I cant believe their mother did not communicate to him that she was bringing them. I just cant take it. I ask him to watch our daughter its always "no shes to little ". I just dont see how he does whatever the other kids mother say, but whatever I ask something he always come up with an excuse. Any advice.