Hallo,
the law doesn't provide a rule that one can't be deported in case of studing if the asylum is fully rejected. That means: legally it is possible. You won't have a residence permit as a student (§16 Aufenthaltsgesetz) after rejected asylum without leaving Germany and making up a visum procedure for studing. But you can ask for a discretionary Duldung (Ermessensduldung) because of an important personal reason (§60.2.3 AufenthG). If you are successful in your studies (especially if you study something economically "useful", with a good perspective to find a job) and your identity is cleared (you've already provided documents, especially a passport, or cooperate with the Ausländerbehörde and do serious steps to provide them), then you have a good chance to get this Duldung. After finishing your studies with a degree you can get a residence permit, if you find a job directly corresponding to your studing profile.
If the Ausländerbehörde doesn't agree to give you this Duldung and insists of your leaving Germany, you can discuss with them that they give you a confirmation of agreement (so called Vorabzustimmung) to your coming back if you agree to go and make a visum procedure in your country. If you give this document in the embassy, you have a good chance that the procedure doesn't take long.