Edited to add (again): as background: I'm a stay at home mom who homeschools. My husband works on oil rigs so when he's gone (4+ weeks), my sister leaves around the same time as everyone else, which is around 7:00. When he's home, he doesn't have to leave during the day for work so we often work separately around the house (me inside and him out) or run errands with the kids (and these are the times my sister stays super late hanging out).

Edited to add: I have addressed my concerns with my husband and we've had many disagreements about it. My sister was involved in one of them several years ago so they're both well aware of my feelings but nothing has changed. Before my family started coming over to our house (before my mom died a few years ago) we would all go over to my parents house. The same situation would occur where they would go and sit outside to hang out while I stayed inside with the kids (although that's not to say that the kids don't go in and out every so often and interrupt). It's also bothersome to me that my husband stays outside until my sister leaves (whether the time is 9:00pm or 2:00 am) and then almost always goes immediately to bed because he's tired.

My husband and I have been married for 14 years and we have 5 kids together. My dad and all four siblings live within minutes of us and since I'm the only ones with kids and a spouse, they all come over every night for dinner. My husband spends his afternoons and nights (until he goes to bed) outside, working on little projects or watching tv/listening to music. One of my sisters (41, single) goes and hangs out with him every night after dinner until she leaves. This usually lasts for at least 4 to 5 hours, but has gone into 8 to 9 hours sometimes. They spend more quality time together than we ever could, because of our kids (aged 13 to 3). My husband acts like it's not weird and there's nothing wrong with it but I think choosing to spend that amount of time with anyone (especially of the opposite sex) who isn't your significant other is incredibly inappropriate. Am I the asshole here?