You point to the agreement. That is what it is for.
In my custom cake and cupcake agreements, every design and flavor change has to be locked 72 hours before the event. After that, a change can be denied or carry a rush change fee. On a wedding cake, changes go in writing by a deadline on the contract.
It holds because the client signed it before you bought a single ingredient. The deposit, 50 percent on a custom cake and 30 percent on a wedding cake in my agreements, covers what you have already ordered. Without a signed window, “can you just change the colors” is a negotiation, and you have no leverage in it.
If you have no agreement yet, write the window into the next one. You set the fee, so decide the number ahead of time, not on the phone.
This is how my own agreements work, not legal advice. Check what applies where you bake.