mibac mibac - 10 days ago 4
Java Question

Deadlock caused by method requiring threads

I'm currently having trouble figuring out the proper way to do this.

I have a ExecutorService with a fixed thread pool of 64. I'm requesting to download a kind of Book (one at a time). To download a book I need to: download the book info, download page info and then download a part of the book. When I request to download a book I get every page info and in the same method I download those little parts of the book. The problem is downloading those little parts of books is also done asynchronously (requiring another thread) but at the time all of 64 threads are occupied by the page-downloading threads. I've came up with either adding another ExecutorService or lifting the thread pool to some bigger number like 256. But that just don't feels quite right. Do I have any other options?

Summary of steps and location of the problem:


  1. download book info

  2. download page:


    • page info

    • page part by part -- Deadlock - out of threads.

      @Override
      public Book getBook(int bookId) {
      Book book = books.get(bookId);
      if (book == null) {
      HttpURLConnection conn = factory.getBook(bookId);
      String s = read(conn);
      book = interpret.readBook(s);

      books.put(book.getId(), book);
      }

      return book;
      }

      @Override
      public Page getPage(int bookId, int pageNum) {
      String s = read(factory.getPage(bookId, pageNum));
      List<Integer> eIds = interpret.readExercises(s);
      List<Exercise> exercises = new ArrayList<>(eIds.size());
      CountDownLatch latch = new CountDownLatch(eIds.size());

      System.out.println("D: Requesting to dl page " + bookId + '>' + pageNum);
      for (int eId : eIds) {
      System.out.println("eId" + eId);
      service.submit(() -> {
      try {
      // The code here does not execute to the lack of free threads
      System.out.println("D: Requesting to dl exer " + eId);
      String sE = read(factory.getExercise(bookId, eId));
      Exercise exercise = interpret.readExercise(sE);
      exercises.add(exercise);
      latch.countDown();
      } catch (Exception e) {
      e.printStackTrace();
      }
      });
      }

      try {
      latch.await();
      } catch (InterruptedException e) {
      e.printStackTrace();
      }

      return new Page(pageNum, exercises);
      }

      @Override
      public WholeBook getWholeBook(int bookId) {
      Book book = getBook(bookId);
      List<Page> pages = new ArrayList<>(book.getPages().size());
      CountDownLatch latch = new CountDownLatch(book.getPages().size());
      System.out.println("D: Requesting to dl book " + bookId);
      for (int pageNum : book.getPages()) {
      service.submit(() -> {
      try {
      Page page = getPage(bookId, pageNum);
      System.out.println("Got page: " + page);
      pages.add(page);
      latch.countDown();
      } catch (Exception e) {
      e.printStackTrace();
      }
      });
      }

      try {
      System.out.println("Waiting for book " + bookId);
      latch.await();
      } catch (InterruptedException e) {
      e.printStackTrace();
      return null; // Better to return null rather than corrupted data
      }

      return new WholeBook(book, pages);
      }




The end of the output is:

D: Requesting to dl page 10753>67
eId235082
eId235092

After that it stops (is technically running but not doing anything)

When I interrupt the thread (using debugger) stack trace points to #getPage and to be more exact to
latch.await()
.

Answer

Since you're executing two different kinds of tasks, and the second one is a subtask of the first one, you end up with the executor being full of the first tasks, which can't finish since their subtasks can't execute. While this is not a classic example of a deadlock, I'd say it qualifies.

The way I would handle this is by removing the use of the executor in getPage(). If for some reason (although I fail to see any valid reason) you want/need to keep the getPage() using multiple threads, you'll have to provide a separate Executor for it to use, so the subtasks will always have a chance to finish.