Figuring out me - unapologetically?

I can't be the only person who struggles with this.
How do you work out who you are, and what you want, without feeling like you need to apologise or add qualifiers or explain your thought processes so that you don't offend people?

I'm really struggling to work this out.
Example.
My family are coming up to Birmingham this weekend to celebrate my birthday with me. We're going to Cadbury World.
Do I want to go to Cadbury World? No, not really. I went a few years ago with friends from college and it was fun, but I don't think it's changed that much.
So why are we going to Cadbury World? Because when we were in Edinburgh, my cousin's girlfriend suggested it.
Well, if you didn't want to go, why did you agree?
Excellent question. The answer? Because I feel guilty about everyone travelling all the way up to Birmingham, just to see me. My dad will drive approximately 400 miles in one day, just to spend some time with me, and I figure if everyone is going to make such an effort, they should get to do something fun. Enter Cadbury World.
Of course, going to Cadbury World adds another layer of guilt. It's costing people about £17 each to go. On top of a meal out. So I feel guilty about people travelling up to see me, and agree to a fairly expensive day trip, and then feel guilty about them having to spend so much money, just to visit me.

Logically speaking, that's madness, right?
It would make so much more sense to have said no to Cadbury World, so I'd only feel guilty about them travelling up to see me.
Why do I feel guilty about them coming up to see me?
Because it's something I can't often do in return. They meet up every other month or so, for birthdays and Easter and so on, and I can't usually make it. This is for a variety of reasons:
My anxiety makes it difficult to travel.
I'm a carer at home, and don't really have a back up, so have to leave my mum on her own.
I'm not really on anyone's way to my Grandad's place in Feltham, so have to make my own way down. Train tickets cost a lot, and coach travel takes forever.

I do this a lot.
Feel guilty over things that I don't have a lot of control over, and then overcompensate by doing things I don't want to do, or going places I don't want to go to assuage my guilt. I then feel guilty for the added hassle that the thing or place causes.

How do I stop feeling guilty for everything?
That's a genuine question. This pattern of behaviour is one I replicate a lot, because feeling guilty for things, even things out of my control, is something I'm used to.
So how do I get un-used to it?
How do I figure out what I want, and not feel guilty, or like and inconvenience, or a burden?
How do I work out how to say what I want, politely but firmly, and stick to it?

Because this birthday thing? It's never going away. I have a birthday every year, and it's going to keep coming up.
Wanting to spend some time on my own at home, is something that's probably going to continue. I would have no problem with my mum or sister wanting to hang out on their own for a while, so why do I feel guilty for wanting to do the same thing?
Wanting to watch a certain TV programme shouldn't mean only watching it when everyone else is asleep or out of the room, so that I don't bother them with what I want to watch. I mean, sure, some stuff people aren't interested in, and being sensitive to that just makes me not a dickhead. But watching my TV with the volume down low, when I think people are asleep so that I don't subject them to my stuff isn't normal. And it's not something that other people feel the need to do, is it?

I would love to understand why my guilt button is so overinflated.
Is this a female thing? I've read articles stating that women feel the need to apologise more than men, that we're brought up to trivialise our thoughts and feelings, and instead focus on everyone else.
Is it an anxiety thing? That would make a lot of sense. A bitch of a mental illness, convincing you that your wants are an incovenience to other people and that you should just thank your lucky stars that they even want to be around you.
But even if you work out the root cause, how do you work around that? How do you unlearn that negative thought process and start living a more healthy one?

Answers on the back of a postcard, please...