What would you do about this situation?

Well me and my child's father were never together just a benefit thing. We didn't get along through much of my pregnancy. Well we barely even talked. Now we are becoming friends but he keeps getting mad at because I won't allow our little girl(4 months old) come visit him overnight. He has only seen her 6 times since she was born. I told him that I would have to meet his family and knows where he stays before I just completely allow her to stay the night with him. I've met his mom, one of his sister and one of his brothers. I just don't feel comfortable sending her off over night to be with well strangers cause that's basically who they are strangers. He stopped visiting after I told him the first time that she couldn't stay overnight. I've tried explaining myself which I feel I shouldn't have too but he gets mad at me anyway for days and then he tries to act like nothing is wrong. I hate our relationship with a pion. We barely talk now, we're in love with each other but we both want to act as though we want someone else. He's spent more time with me then the baby and I feel bad because I feel as though I'm taking away from her. We attend the same school by the way and stay an hour or so away from each other. I wanna just drop my boyfriend to be with my bd but I'm afraid of being walked out on again. I'm happier with my bd than my boyfriend. I know I'd be even more happier if our child was with us while we are together. What should I do???