From Dhaka:

If you choose the Bus, that goes to Sylhet. On the Putijuri Bazar you can depart from the bus and choose CNG (three wheeler) to go to directly to The Palace.

If you choose the Train, Go for Shaistaganj via train. Then take a CNG (three wheeler) to go to directly to The Palace.