Welcome to the Wefugees Community and thank you for reaching out to us.
I did not fully understand yet what kind of white paper the immigration office is issuing regularly - is it a "Grenzübertrittsbescheinigung (GüB)"?
We discussed a very similar question on this platfom before - please see this thread for more information on residence permits for EU citizens' spouses. As you are from Ghana, a so-called "safe country of origin", please have also a closer look at my colleague Alla's explanation below.
In any case, don't hesitate to get back to us with any further questions.
All the best,