You Don't Owe Them Anything
Just as it takes a village to raise a child, we are formed in our life by many people. Some of these people may feel entitled to your time and attention because they "invested" in you. You don't owe them anything. This doesn't mean you must be resentful toward them or their opinions, quite the opposite. You can be grateful to who they have been to you and have positively contributed to you as a person.
Even relationships that were entirely healthy and benevolent may reach a change of course when people choose different life paths. You do not need to let yourself be dragged into the past or "submit to authority." Trust yourself to make good choices and be free to engage or not. Again, you don't owe them anything.