Options for ValidationSet accuracy from NetTrain?

I think for the ValidationSet -> Scaled[.1] the validation will be sampled with randomly from the training data. If we want to control the sample ourselves, we can split them manually and supply it as ValidationSet -> testData.

We can use the TrainingProgressFunction function to get more information about the validation accuracy, and abort the training after a certain goal is achieved.

For example, this will stop the training process once the validation loss is smaller than 0.1:

resource = ResourceObject["MNIST"];
trainingData = 
  RandomSample[ResourceData[resource, "TrainingData"], 6000];
testData = RandomSample[ResourceData[resource, "TestData"], 1000];
lenet = NetChain[{
    ConvolutionLayer[20, 5], Ramp, PoolingLayer[2, 2],
    ConvolutionLayer[50, 5], Ramp, PoolingLayer[2, 2],
    FlattenLayer[], 500, Ramp, 10, SoftmaxLayer[]},
   "Output" -> NetDecoder[{"Class", Range[0, 9]}],
   "Input" -> NetEncoder[{"Image", {28, 28}, "Grayscale"}]
   ];

loss = {};
trained = 
 NetTrain[lenet, trainingData, ValidationSet -> testData, 
  MaxTrainingRounds -> 20, 
  TrainingProgressFunction -> {AppendTo[loss, #ValidationLoss]; 
     If[#ValidationLoss < 0.1, Abort[]]; &, 
    "Interval" -> Quantity[1, "Rounds"]}]

loss
(* {None, 0.474543, 0.290683, 0.174869, 0.146322, 0.110496, 0.109694, 0.0923467} *)

I'll add to our ToDo list the idea of having the validation indices be available as a property of NetTrain.