I’ve recently found it too that it works this way. The reason I think is that the only way we can allow mic usage to the app is while it’s in use (“allow only while using the app”).

maybe the “ask every time” option would not have this limitation. but it’s the developers who could fix it, probably by changing how the permission is requested