We consider a class of optimization problems in which the aim is to find some optimizing probability distributions. One particular example is optimal design. We first review the optimal design theory, and determine the optimality conditions using directional derivatives. We then construct optimal designs for various polynomial regression models by finding the analytic solutions and by using a class of algorithms. We consider a practical problem, namely a radiation-dosage example, and discuss important aspects of optimal design throughout this example. We also construct optimal designs for various polynomial regression models with more than one design variable. We consider another practical problem, namely a vocabulary-growth study. We then construct D-optimal and c-optimal designs for various models with and without the interaction term and the second order terms in design variables. We also develop strategies for constructing designs by using the properties of the directional derivatives.