Being you don't have a child yourself, it is hard to put yourself, in HIS shoes. HE has a child. THAT is his 1st priority. AND he has to abide by his custody orders and his custody responsibility.
It would be a REAL shame... if your Boyfriend, for example, due to you wanting more "alone" time with him... had to, hire a babysitter to watch his daughter, while you both go out on a date. On HIS... weekend with his daughter. That is... totally sabotaging his time/custody arrangement/parenting time, with his daughter. And perhaps, I imagine, her Mom would NOT be happy with that.
She is a child.
You perhaps, can read up on child development. If it is important, to you.
Kids, are not just one emotion, all the time. Nor are they "happy" all the time. And at 3.5 years old, they are not even fully developed yet. A teenager is not even fully developed yet and they have TONS more hot/cold emotions than a 3 year old. For example.
Your priority is, your Boyfriend. And being alone with him, not having to share him with a mere child.
Well, he has a child. And that is the constant everyday reality, you MUST deal with. Or not. It is your, choice.
His child, is a priority. THE priority. And all the laws surrounding custody and with his Ex.
A child, his child... is not something you can just shoo away. Like a mosquito. It is a 24/7, duty. Whether you like it or not. That is called, parenting.
And if your Boyfriend ever feels... forced by you...to disregard his daughter, for you, well that is going to be a VERY dysfunctional... vicious cycle... that is imposed on him... AND his child. AND it will... adversely... affect his daughter and her emotional development. And her relationship... with her Dad.
This relationship is not up to you. A parent, cannot and should not have to choose, between you or their child.
ALSO: how do you view yourself and your impact upon this child, even if it is someone elses child???? Because, just by being "Girlfriend" to the Dad... you ARE affecting this child. For good or bad.
And you are not this child's Mom.
Again, if you want weekends alone with him, this will NOT happen. Because, your Boyfriend is a Dad. And the weekends are when HE has his daughter. WITH him. And this time with his Daughter, is not to be wasted. Why should he.... not spend time with his daughter on his weekends with her? He is a parent. This is how it is.
And, no child should grow up, thinking that their parent just wants to get away from them... or feel that they are in the way.
Now: per what you said about feeling "isolated...." and "in a difficult position...." and not having any alone time with your Boyfriend on the weekends to "build our relationship..." "based on our interests...."
WELL:
how do you think, a 3.5 year old... feels about that TOO???????????
Can you imagine, his daughter feeling JUST like you... ??????
No child, should feel that way with her Dad.
But, you are an adult, a Girlfriend. So YOU should be the one... to be more mature about it and think... about the consequences of your actions and relationship, upon this child.
Because, the child, is caught in the middle.
And PROBABLY, his daughter wishes SHE HAD ALONE TIME only with her Dad. AND probably wishes SHE can build her relationship with her Dad and "connect with him..." being she only sees him on weekends.
But she can't.
You are there.
His daughter, does not get to see her Dad, during the week like you can. His daughter can only see him on weekends.
And his daughter, did NOT know... what she was getting into as a child with a Dad...who can only see her on weekends. She did not have a say in it.
But you know, your Boyfriend is a Dad, and has a child.
And she is "isolated" only seeing her Dad on the weekends.
A little girl, needs their Dad, if he is even in the picture.
And he is, in her picture and life.
And she is "already in a difficult position...." having lost, her parents due to divorce. But she is a child. And all their lives, a child NEEDS their Dad and Mom.
Why can't you see your Boyfriend during the week?
And worst case scenario:
That your Boyfriend, starts to imagine, that his Girlfriend is more important than his daughter and his own time, with his daughter.
Hopefully he advocates for his daughter and nurtures, her. His child.