The "brown part" is called your turbine housing, exhaust side, or hot side. The other side is your compressor side.

If you have cracks in your exhaust side of the turbo then you may (depending on the size of the cracks) experience increased turbo lag due to exhaust leaks. If they are big enough then you may not get boost at all. I think cracks like this can be welded back together, but I'm not totally sure. IMO, buy a new exhaust side.