Which transformations will produce similar, but not congruent, figure

Rotations, translations and reflections are transformations that preserve areas, so they always produce congruent figures.
So, the only way you have to produce a similar but not congruent figure is to use dilation with a scale different than 1.