Hi @Gerabest123
Firstly, as you have a lawyer he is he best person to be able to advise you on this. We are not lawyers and so I can only provide a general evaluation of your situation.
Some things are not clear to me in your question. You state your application was rejected due to a criminal offence. An application for residence on family grounds can only be rejected for serious criminal offences. If the offence is illegal entry to Germany or illegal overstay of your visa conditions, then these are not sufficiently serious offences.
It is also not clear why the rejection was not appealed and instead a letter was sent asking for a Duldung.
You child is entitled to a residence permit according to paragraph 33 of the residency law (Aufenthaltsgesetz).
You may be entitled to a residence permit due to your husband. He needs to earn enough to support himself, you and the baby financially. You will also have to show you speak A1 level of German. There are some exception to the German language rule, depending on what type of permit he had before getting permanent residency.
If you fulfil the requirements for a permit on family grounds, the authorities will most likely tell you to first leave Germany and apply for a visa for family reunification at a German embassy. As you have a baby, however, it is not reasonable for you to have to do this. It would require your child to be separated from you for possibly quite some time and in such cases the immigration office (Ausländerbehörde) should process your application here in Germany.
If the visa becomes an issue, it is important that your lawyer is aware of a recent case at the highest German court (Bundesverfassungsgericht). The circumstances of the case are different but the central question is whether it is reasonable to expect someone with a child to leave them to go and get a visa. The court ruled in favour of the applicant for a resident permit. Here is a brief report on it in English via online translation:
https://***-migazin-de.translate.goog/2023/12/04/bundesverfassungsgericht-auslaenderbehoerde-darf-visumpflicht-fuer-familiennachzug-nicht-ueberspannen/?_x_tr_sl=de&_x_tr_tl=en&_x_tr_hl=en&_x_tr_pto=wapp
Best,
Éanna