Wednesday 15 September 2010

Batch files processing in bash with full processor occupancy -


Probably a simple question, but I do not know where to dig, I have a list of files (random name),

And I want to use them with some command

  processing_command $ i $ {i% *}. Txt   

I want to sharpen all the processors using how such a script is captured at 10 processors simultaneously (processing 10 files)? Processing_command is not parallel by default Thank you!

You must use a trivial approach:

  $ items in i Processing_directory $ i $ {i%} *}. Txt & amp; Which will start a new (parallel instance)  processing_command  for every $ i (this trailing is the previous   

which will be the background process), that if your The nearest example is 1000 objects, so it will start 1000 parallel processes, which will be busy in switching context instead of actual processing (while capturing all 10 cores). If you have many (or fewer) items in the form of a coat, then this is a good and simple solution.

Usually you do not want to start more processes than the core.

A Simplified approach (assuming all items take about the same time at the time of processing) is to divide the original "item" list into the same long list as number_of_cores . The following is an example taken from an example of article in German:

  #! / Bin / bash # # Number of processors PMAX = $ (ls-1d / sys / devices / system / cpu / cpu [0-9] * | wc -l) ## processing_command on each argument: doSequential () {Local I In "$ @" for; Processing_directory $ i $ {i%} *}. Txt done} ## PMAX parallel processes doParallel () {# PMMA debates in the size equally locks the items in the item "$ @" in the item current process = 0; Item [$ current process] = "$ {item [$ current process]}" $ item "" go current process = $ (((current process + 1% PMX)) # PMX processes completed, with short list of each item current process = 0 while doing [$ current process -lt $ pmx]; [-n "$ {item [$ current process]}"] & amp; amp; Eval doSequential $ {items [$ currentProcess]} & current process = $ ((current process + 1)) waited} $ DEPRELLE $ item    

No comments:

Post a Comment