I have been having some trouble doing this and was wondering if it is a common theme or there is room to make it better.
Sometimes packages wont compile at all and thats fine. But recently I cross compiled some system packages and it bricked my system and was no longer able to ssh into it. I am not asking for help in this specific issue, but want to ask if this is a common occurrence for this kind of setup? It does not seem to be a popular setup
In general there should not be any problem with most popular build systems (autoconf, cmake etc.) if they are used properly. However some pieces of software use build systems that are not so robust, or even self written scripts to build that causes tons of troubles. E. g. cross compiling samba is very painful. The most difficult case in when some tool is compiled and then used to generate code that will be compiled later, if developers did not implement bootstrapping properly.
So it is impossible to give a common answer. In general cross compiling is not a problem, but in some particular cases you may need to put in a lot of effort to get it working correctly.