Monday 15 July 2013

java - Read the 30Million user id's one by one from the big file -


I am trying to read a very large file using Java. That large file will contain data like this, which means that each line will have a user ID.

  149905320 1165665384 66969324 886633368 1145241312 286585320 1008665352   

And in that large file there will be around 30 million user IDs now that user id once that file I am trying to read once. Which means that each user ID should be selected only from that large file. For example, if I have 3 million user user ID then it should print only 30 million user IDs with the use of the much anticipated code.

Below I have a code which is a multi-thread code running with 10 threads, the program given below, I am not able to ensure that every user ID is selected only once. Public category ReadingFile {public static zero main (string [] args) {// Create a Thread Pool Executable service service = executable with the given size. NewfixedThreadPool (10); For (int i = 0; i

You can not really improve reading a thread in the file sequentially, assuming it That you have not done anything like a file in multiple disks, with a thread, you do a search and then read a long sequential; with many threads you are going to create many threads, Each advantage is known as control.

EDIT: This is a way to parallel line processing while still using serial I / O to read lines It uses one to communicate between the thread; Adds the line to the file task line, and reads them and executes them as CPCTUSC . This is a thread-safe data structure, so there is no need to add any synchronization to it. You are using put (ee) to add strings in the queue, so if the queue is full (it can hold 200 strings, like ReadingFile ) File Task blocks space free; Similarly you are using LE to extract items from the queue, so the code will be blocked by CPUTask until it becomes available.

  public class readings {public static zero main (string [] args) {final integer thread = 10; // BlockingQueue with 200 Blocking Quantity & lt; String & gt; Queue = new Arrawlocking Q & lt; & Gt; (200); // Create a Thread Pool with the given size Executable Services Service = Exporter. NewFixedThreadPool (ThreadCount); For (Int i = 0; I & lt; (ThreadCount1); I ++) {service.submit (new CPUsq (queue)); } // File Wait for the cookies to complete. Service.submit (new file task (queue)) is complete. Service.shutdownNow (); // Stop CPutasks // terminate the CPUTasks service. Wait (365, TimeUnit.DAYS); }} Class file executes the runleble {Private Final Blocking Q & lt; String & gt; Queue; Public file task (BlockingQueue  line) {this.queue = queue; } @ Override Public Wide Run () {Buffered Redirector BR = Faucet; Try {br = new BufferedReader (new FileReader ("D: /abc.txt")); String line; While ((line = br.readline ())! = Null) {// block if queue is the full queue.It is (line); }} Hold (FileNotFoundException e) {e.printStackTrace (); } Hold (IOException e) {e.printStackTrace (); } Finally {try {br.close ()}} hold (IOException e) {e.printStackTrace (); }}}} Class CPUTask implies Runnabel {Private Final Blocking Q & lt; String & gt; Queue; Public SeatsSock (blocking Q & lt; string & gt; queue) {this.queue = queue; } @ Override Public Wide Run () {String Line; While {true} {try {// block if queue is empty line = queue.take ();) // works with line} grip (interrupted preprator pre) {break; // FileTask is complete}} // poll () If empty, then the row is empty ((line = queue.poll ())! = Null) {// works with line; }}}    

No comments:

Post a Comment