Dear @Musawvir ,
Based on my understanding of your situation, you are entitled to a permit based on having been married to an EU citizen for at least three years and at least one of these years in Germany. For such cases, if you are working, you retain your residency rights after your family members leave Germany.
The relevant legal basis for this is paragraph 3, section 4, part 1 of the freedom of movement law (here via online translation):
https://***.gesetze-im-internet.de/freiz_gg_eu_2004/__3.html
Permanent residency is generally only possible after 5 years. In some cases, for example if you are working in a skilled profession, the time required is less.
Best,
Éanna