Welcome back to the Wefugees Community and thank you for your question.
First of all, I think it is a very big step and a positive development that the immigration office issued a residence permit according to section 25b of the German Residence Act after you found yourself in a limbo for several years. This is something you can build on. The "Niederlassungserlaubnis" (permanent residence permit), however, requires that people hold a temporary residence permit for at least 5 years. I found a very nice overview which depicts the steps well.
For a full list of the requirements for the Niederlassungserlaubnis you may also follow this link, for instance (here exemplary for Berlin residents).
I hope this helps and please don't hesitate to get back to us with any further questions.
All the best,