I am far from a fashionista, so I won't tell you what to wear or what color (although pink is a good bet for Oaks), but I can tell you how the weather may affect your choice of clothing.
Layers may be a good option with very cool/chilly mornings and cool/mild afternoons. Kentucky Oaks Friday will be the cooler day with 40-50s in the morning, and low 60s in the afternoon with a partly sunny sky. A very slight chance of an afternoon shower. So small, I wouldn't stress if you forget the rain gear.
The Kentucky Derby forecast is improving with another cool morning, but a milder afternoon with a high near 70. Sunshine on Saturday always helps make it feel warmer out at the track as well.
One thing we haven't mentioned is wind. Winds won't be too strong, but breezy at times. Expect wind gusts 10-20 mph. While the wind won't be a big bother, it might add a little chill factor, especially in the morning.
Good luck picking out the perfect clothes and enjoy some pretty good weather for Oaks and Derby!
Here's a look at each Dayplanner. 8am: Gates Open. 10:30am: First Post. 5pm: High temp.