What features might non trees but with same ecological functions of trees, take?
Well, the ocean provides examples of some. Kelp forests and corals are good examples of things that grow big and are looking for light. Both have tree like habits and growth in different ways
What do you mean by ecological functions? Surviving and reproducing?
You know how coral polyps have a mobile stage before they settle down and turn into coral in a reef? What if a tree had a young stage where it was like a bird or a deer or something? The tree grows a few great big fruits each year (or perhaps saves up energy to fruit once every several years) and when the fruit is ripe it cracks open on the ground to let out an animal. This animal then either travels some distance and plants itself and becomes a sapling if the tree is an R strategist or if it is a K strategist then it feeds the animal a milk like sap until it can hunt on its own and then the animal goes out to do animal things and get big and fat and then plants itself and becomes a big strong sapling. Both strategies would help disperse the plant far and be better at finding good locations than traditional seeds, and the K strategy would also serve to help the young not compete with the old for resources (old eat lots of sunlight and canopy space, young eat roots and tubers and stuff from other plants) and possibly also help recover quickly after a storm (big trees go down in strong winds but it triggers the young tree-deer to plant themselves and they outcompete other species trying to take that space) or to help with parasites or other competitors (young tree-birds spend the first year of their life eating aphids and caterpillars off the tree, young tree-deer might dig up saplings of other species and plant themselves in their place).
At extremes of this kind of evolutionary path you might even get eusocial and cast behavior like you see with ants. Each tree might be protected by a flock of tree-birds who protect the tree and maybe forage for it to suplament its roots and leaves and the tree might feed and shelter the tree-birds during the winter. Then every once in a while the tree sends out a drone tree-bird to go find and mate with another tree’s queen tree-bird who will then plant herself and repeat the cycle.
The design of a tree is likely to be quite convergent. E.g. trees and palm trees evolved the same basic layout.
Light trapping organisms require as much surface area exposed to the local star as they can cope with. Trees evolved by plants trying to climb over one another. It took the evolution of cellulose for them to get really tall. Alien plans might now have evolved an equivalent.
The leaf structure, on earth is a trade-off. On earth, it needs sunlight, water from the ground and CO2 from the air. It also needs to deal with damage. A lot of holes is great for CO2 absorption, but will rapidly lose water. A flat surface is great for catching the sun, but leaves all the expensive machinery exposed to damage. Balancing all of these leads to the various leaf shapes we see on both trees and smaller plants.
There’s still room for more interesting ideas, but they are unlikely to be the staples of an alien forest. E.g. if the air composition was different, balloon plants might be viable. Rather than a hard, woody stem, the top could be a hydrogen filled leaf balloon. A vine could bring water up from the roots. It would act like a lilly pad does in water.