If you add more colour to a scene, there's less you can do with light/shadow because you're giving up some of the RGB range to show the colour.

White light will always create more contrast compared to other colours.