Welcome to Wefugees and Congratulations on the addition to your family! :-)
I'm sorry to hear you are experiencing these difficulties with your residence permit application. From what I could find on our platform, it is not uncommon for German authorities to ask spouses to return to their "home country" and come back on a spouse visa/ family reunion visa.
It's a good idea to let a refugee support service have a closer look at your case and help you with this urgent matter in person!
Here are the contact details of the Refugee Law Clinic in Nuremberg: http://www.rlc-erlangen.de/beratung.html
You can find their phone number and E-Mail on there to reach out to them.
Otherwise, there is a list of lawyers specializing in asylum law here: https://www.anwalt.de/verzeichnis/rechtsanwaelte.php?stadt=Nuernberg&rechtsgebiet=Auslaenderrecht_und_Asylrecht&cityid=12888
And a list of other pro bono legal aid services for refugees in Bavaria: http://www.refugeelegalaidinformation.org/bayern-bavaria-pro-bono-directory
I hope these contact details help and you find support with your case quickly. Feel free to reach out to us again if you need further assistance with your case!
All the best and Good Luck,