How to make certain number of threads running all the time
Personally I would use PLINQ for this, and specifically the WithDegreeOfParallelism method which limits the number of concurrent executions to the passed in value.
private IEnumerable<Action> InfiniteFunctions()
{
while(true)
{
yield return func_myTask;
}
}
private void Button_Click_4(object sender, RoutedEventArgs e)
{
int irMaximumThreadcount = 100;
InfiniteFunctions()
.AsParallel()
.WithDegreeOfParallelism(irMaximumThreadcount)
.ForAll(f => f());
}
EDIT: Actually reading the documentation it seems that irMaximumThreadCount can only be a max of 64 so watch out for that.
EDIT 2: Ok, had a better look and it seems Parallel.ForEach
takes a ParallelOptions
parameter which includes a MaxDegreeOfParallelism
property that isn't limited - Check it out. So your code might be like:
private void CrawlWebsite(string url)
{
//Implementation here
}
private void Button_Click_4(object sender, RoutedEventArgs e)
{
var options = new ParallelOptions()
{
MaxDegreeOfParallelism = 2000
};
Parallel.ForEach(massiveListOfUrls, options, CrawlWebsite);
}
You are mixing up tasks with threads. A task is not a thread. There is no guarantee that each task will have it's own thread.
Actually the TPL (Task Parallel Library) is some kind of queue. This means you can just create and start tasks for each Func
or Action
object you have. There is no easy way to control the number of threads that are actually created.
However, you can create many tasks with little overhead because the TPL will enqueue them and apply further logic to balance the work over the threads of the thread pool.
If some tasks need to be executed one after the other you can use Task.ContinueWith
to enqueue them. It is also possible to start new tasks with Task.Factory.ContinueWhenAny
or Task.Factory.ContinueWhenAll
.
This is also the clue to how you can control the number of parallel tasks you want to create: Just create the desired number of tasks and enqueue the remaining tasks with ContinueWhenAny
. Each time a task ends the next will be started.
Again: the TPL will balance the work among the threads in the thread pool. What you need to consider anyway is the use of other resources like disk I/O or internet connection. Having a lot of tasks that try to use the same resources concurrently can drastically slow down your program.