The ability to say no comes from self-respect (not worth my time) and self-awareness (not something I can do).

If you find it hard to say no, you lack either or both.