It took some time to understand the config file (which is a general issue with linux apps), since mediatomb can stream divx movies/mp3 files without transcoding, getting transcoding to work with vlc was trublesome, there are 4 settings that have to match.
First the mime type reported to the ps3 which only seems to accept video/mpeg.
Second the video codec of the transcoded stream, only mpgv seems to be accepted.
Third the audio stream had to be mpga, mp3 doesnt seem to work
Finally the container of the streamed transcoded media which is ps
That's great because i could read that from most tutorials/blogs.
But to get transcoding to actually work the section of mediatomb config file had to map to a profile that will transcode the file eg:
Here is my full config file: config.xml
And my transcode scripts (using vlc)
For mkv files: mediatomb-transcode-video
For dvd/iso images: mediatomb-transcode-video-iso