Congratulations if you are invited ! Because French people take their time but once you are invited, it means they want to know more about you, and maybe to introduce you to their private circle. And that's important, because in France, people don't have a lot of friends, they have a few but close friends. So if you are invited, it might be the beginning or a deep friendship. At least, it means that French person had enough time to make up his mind and decide it was worth trying.
So if you agree say “oui, merci, c'est gentil”, “avec plaisir”
If you can't or don't want, say “Désolé mais je ne peux pas”, “c'est gentil mais je ne peux pas”, “ça me fait plaisir mais je ne peux pas”.
Back to je suis invité / I'm invited