Mr_RexZ Mr_RexZ - 1 month ago 23
Java Question

Worker process cannot be executed in a Java Tomcat server deployed to Heroku

I have 1 web and 1 worker process communicating with each other using

CloudAMQP
library consisting of 2 channels/queues (1 channel is to send a simple message from
web
, and the other channel is to receive a message from
worker
). Since I'm deploying my webapp in
WAR
file format, I had to put my
worker
class into a jar by creating a local repository of Maven and including dependency to put the file in
WEB-INF/lib
folder so that I could execute the worker process using the generated
worker sh
script from
maven-war-plugin
artifact.

worker sh
script :


BASEDIR=`dirname $0`/..
BASEDIR=`(cd "$BASEDIR"; pwd)`



# OS specific support. $var _must_ be set to either true or false.
cygwin=false;
darwin=false;
case "`uname`" in
CYGWIN*) cygwin=true ;;
Darwin*) darwin=true
if [ -z "$JAVA_VERSION" ] ; then
JAVA_VERSION="CurrentJDK"
else
echo "Using Java version: $JAVA_VERSION"
fi
if [ -z "$JAVA_HOME" ] ; then
JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Versions/${JAVA_VERSION}/Home
fi
;;
esac

if [ -z "$JAVA_HOME" ] ; then
if [ -r /etc/gentoo-release ] ; then
JAVA_HOME=`java-config --jre-home`
fi
fi

# For Cygwin, ensure paths are in UNIX format before anything is touched
if $cygwin ; then
[ -n "$JAVA_HOME" ] && JAVA_HOME=`cygpath --unix "$JAVA_HOME"`
[ -n "$CLASSPATH" ] && CLASSPATH=`cygpath --path --unix "$CLASSPATH"`
fi

# If a specific java binary isn't specified search for the standard 'java' binary
if [ -z "$JAVACMD" ] ; then
if [ -n "$JAVA_HOME" ] ; then
if [ -x "$JAVA_HOME/jre/sh/java" ] ; then
# IBM's JDK on AIX uses strange locations for the executables
JAVACMD="$JAVA_HOME/jre/sh/java"
else
JAVACMD="$JAVA_HOME/bin/java"
fi
else
JAVACMD=`which java`
fi
fi

if [ ! -x "$JAVACMD" ] ; then
echo "Error: JAVA_HOME is not defined correctly."
echo " We cannot execute $JAVACMD"
exit 1
fi

if [ -z "$REPO" ]
then
REPO="$BASEDIR"/repo
fi

CLASSPATH=$CLASSPATH_PREFIX:"$BASEDIR"/etc:"$REPO"/com/rabbitmq/amqp-client/3.3.4/amqp-client-3.3.4.jar:"$REPO"/javax/servlet/jstl/1.2/jstl-1.2.jar:"$REPO"/com/example/worker/1.0/worker-1.0.jar:"$REPO"/cloudamqp/example/amqpexample/1.0-SNAPSHOT/amqpexample-1.0-SNAPSHOT.war
EXTRA_JVM_ARGUMENTS=""

# For Cygwin, switch paths to Windows format before running java
if $cygwin; then
[ -n "$CLASSPATH" ] && CLASSPATH=`cygpath --path --windows "$CLASSPATH"`
[ -n "$JAVA_HOME" ] && JAVA_HOME=`cygpath --path --windows "$JAVA_HOME"`
[ -n "$HOME" ] && HOME=`cygpath --path --windows "$HOME"`
[ -n "$BASEDIR" ] && BASEDIR=`cygpath --path --windows "$BASEDIR"`
[ -n "$REPO" ] && REPO=`cygpath --path --windows "$REPO"`
fi

exec "$JAVACMD" $JAVA_OPTS \
$EXTRA_JVM_ARGUMENTS \
-classpath "$CLASSPATH" \
-Dapp.name="worker" \
-Dapp.pid="$$" \
-Dapp.repo="$REPO" \
-Dbasedir="$BASEDIR" \
WorkerProcess \
"$@"


The
worker sh
script is then referenced into Heroku
Procfile
:

web: java $JAVA_OPTS -jar target/dependency/webapp-runner.jar --port $PORT target/*.war
worker: sh target/bin/worker


I tested the app locally using
heroku local -f Procfile.windows
, and all works as expected. However, when I deploy to Heroku (I have tried using both Heroku CLI deployment and
git
as well), only the
web
Procfile can be executed properly. The worker process gives back
error 127
when I check the log, saying it cannot open
target\bin\worker
script. I've tried to google what
error 127
for heroku means, but so far there's no match, and I have no idea what causes of my remotely deployed apps not being able to open
target\bin\worker
even though it works fine in my local deployment. Any help will be appreciated.
Below here are the full error logs from Heroku, servlet class, worker class, and POM respectively :

2016-10-31T14:27:32.629768+00:00 heroku[slug-compiler]: Slug compilation finished
2016-10-31T14:27:34.807925+00:00 heroku[web.1]: State changed from down to starting
2016-10-31T14:27:34.941663+00:00 heroku[worker.1]: State changed from down to starting
2016-10-31T14:27:37.334213+00:00 heroku[worker.1]: Starting process with command `sh target/bin/worker`
2016-10-31T14:27:37.825597+00:00 heroku[worker.1]: State changed from starting to up
2016-10-31T14:27:38.100208+00:00 app[worker.1]: sh: 0: Can't open target/bin/worker
2016-10-31T14:27:38.104979+00:00 heroku[worker.1]: State changed from up to crashed
2016-10-31T14:27:38.104979+00:00 heroku[worker.1]: State changed from crashed to starting
2016-10-31T14:27:38.291551+00:00 heroku[web.1]: Starting process with command `java $JAVA_OPTS -jar target/dependency/webapp-runner.jar $WEBAPP_RUNNER_OPTS --port 54481 target/amqpexample-1.0-SNAPSHOT.war`
2016-10-31T14:27:40.725610+00:00 app[web.1]: Setting JAVA_TOOL_OPTIONS defaults based on dyno size. Custom settings will override them.
2016-10-31T14:27:40.732741+00:00 app[web.1]: Picked up JAVA_TOOL_OPTIONS: -Xmx350m -Xss512k -Dfile.encoding=UTF-8
2016-10-31T14:27:41.385588+00:00 app[web.1]: Expanding amqpexample-1.0-SNAPSHOT.war into /app/target/tomcat.54481/webapps/expanded
2016-10-31T14:27:41.385722+00:00 app[web.1]: Adding Context for /app/target/tomcat.54481/webapps/expanded
2016-10-31T14:27:41.817336+00:00 heroku[worker.1]: Starting process with command `sh target/bin/worker`
2016-10-31T14:27:42.172560+00:00 app[web.1]: INFO: Initializing ProtocolHandler ["http-nio-54481"]
2016-10-31T14:27:42.172549+00:00 app[web.1]: Oct 31, 2016 2:27:42 PM org.apache.coyote.AbstractProtocol init
2016-10-31T14:27:42.201060+00:00 app[web.1]: Oct 31, 2016 2:27:42 PM org.apache.tomcat.util.net.NioSelectorPool getSharedSelector
2016-10-31T14:27:42.201064+00:00 app[web.1]: INFO: Using a shared selector for servlet write/read
2016-10-31T14:27:42.205036+00:00 app[web.1]: Oct 31, 2016 2:27:42 PM org.apache.catalina.core.StandardService startInternal
2016-10-31T14:27:42.205038+00:00 app[web.1]: INFO: Starting service Tomcat
2016-10-31T14:27:42.206587+00:00 app[web.1]: Oct 31, 2016 2:27:42 PM org.apache.catalina.core.StandardEngine startInternal
2016-10-31T14:27:42.206588+00:00 app[web.1]: INFO: Starting Servlet Engine: Apache Tomcat/8.0.30
2016-10-31T14:27:42.378327+00:00 heroku[worker.1]: State changed from starting to up
2016-10-31T14:27:42.441771+00:00 app[web.1]: Oct 31, 2016 2:27:42 PM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment
2016-10-31T14:27:42.441781+00:00 app[web.1]: INFO: No global web.xml found
2016-10-31T14:27:42.765321+00:00 heroku[web.1]: State changed from starting to up
2016-10-31T14:27:43.507545+00:00 app[worker.1]: sh: 0: Can't open target/bin/worker
2016-10-31T14:27:43.591348+00:00 heroku[worker.1]: Process exited with status 127
2016-10-31T14:27:43.603552+00:00 heroku[worker.1]: State changed from up to crashed
2016-10-31T14:27:44.410603+00:00 app[web.1]: Oct 31, 2016 2:27:44 PM org.apache.jasper.servlet.TldScanner scanJars
2016-10-31T14:27:44.410621+00:00 app[web.1]: INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
2016-10-31T14:27:44.498715+00:00 app[web.1]: Oct 31, 2016 2:27:44 PM org.apache.coyote.AbstractProtocol start
2016-10-31T14:27:44.498718+00:00 app[web.1]: INFO: Starting ProtocolHandler ["http-nio-54481"]
2016-10-31T14:27:47.410949+00:00 heroku[router]: at=info method=GET path="/" host=amqpexample.herokuapp.com request_id=334f7e05-d4e5-4f78-a444-1e6e7403e52c fwd="14.192.210.168" dyno=web.1 connect=1ms service=4305ms status=200 bytes=370
2016-10-31T14:27:50.003897+00:00 heroku[router]: at=info method=GET path="/" host=amqpexample.herokuapp.com request_id=51c9fac7-6763-4374-9f2b-672f0d7f7be7 fwd="14.192.210.168" dyno=web.1 connect=1ms service=21ms status=200 bytes=295


Servlet class (responsible to a display a buffer message and waiting for new message to be sent from
worker
when being refreshed) :

package herokutest;

import com.rabbitmq.client.*;

import java.io.IOException;
import java.net.URISyntaxException;
import java.security.KeyManagementException;
import java.security.NoSuchAlgorithmException;
import javax.servlet.ServletConfig;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

public class HelloWorld extends javax.servlet.http.HttpServlet {

private final static String QUEUE_NAME = "hello";
private final static String ANOTHER_QUEUE = "ANOTHER";

public Channel channel;
public Channel anotherChannel;
public DefaultConsumer consumer;
public String message = "Temporary";

public void init(final ServletConfig config) throws ServletException {
try {
createChannel();
} catch (Exception e) {
e.printStackTrace();
}
finally {
consumer = new DefaultConsumer(anotherChannel) {
@Override
public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body) throws IOException {
String innerMessage = new String(body, "UTF-8");
message = innerMessage;
}
};
try {
anotherChannel.basicConsume(ANOTHER_QUEUE, true, consumer);
} catch (IOException e) {
e.printStackTrace();
}
}
}

public void createChannel() throws NoSuchAlgorithmException, KeyManagementException, URISyntaxException, IOException {
String uri = System.getenv("CLOUDAMQP_URL");
if (uri == null) uri = "amqp://guest:guest@localhost";

ConnectionFactory factory = new ConnectionFactory();
factory.setUri(uri);
factory.setRequestedHeartbeat(30);
factory.setConnectionTimeout(30);
Connection connection = factory.newConnection();
channel = connection.createChannel();
anotherChannel = connection.createChannel();

channel.queueDeclare(QUEUE_NAME, false, false, false, null);
anotherChannel.queueDeclare(ANOTHER_QUEUE, false, false, false, null);

}


protected void doGet(final HttpServletRequest request, final HttpServletResponse response) throws ServletException, IOException {

String test="test";
channel.basicPublish("", QUEUE_NAME, null, test.getBytes());
request.setAttribute("message", message);
request.getRequestDispatcher("/index.jsp").forward(request, response);

}
}


Worker class : (responsible for sending a message
Another message
back to servlet)

import com.rabbitmq.client.*;

import java.io.IOException;

public class WorkerProcess {
private final static String QUEUE_NAME = "hello";
private final static String ANOTHER_QUEUE = "ANOTHER";
static String message;



public static void main(String[] argv) throws Exception {


ConnectionFactory factory = new ConnectionFactory();

String uri = System.getenv("CLOUDAMQP_URL");
if (uri == null) uri = "amqp://guest:guest@localhost";
factory.setUri(uri);
factory.setRequestedHeartbeat(30);
factory.setConnectionTimeout(30);

Connection connection = factory.newConnection();

Channel channel = connection.createChannel();
final Channel anotherChannel = connection.createChannel();
channel.queueDeclare(QUEUE_NAME, false, false, false, null);
anotherChannel.queueDeclare(ANOTHER_QUEUE, false, false, false, null);
System.out.println(" [*] Connected to Worker");

final Consumer consumer = new DefaultConsumer(channel) {
@Override
public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body) throws IOException {

message = new String(body, "UTF-8");
String anotherMessage= "Another message";
anotherChannel.basicPublish("", ANOTHER_QUEUE, null, anotherMessage.getBytes());
}
};
channel.basicConsume(QUEUE_NAME, true, consumer);

}

}


POM :

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>

<groupId>cloudamqp.example</groupId>
<artifactId>amqpexample</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>war</packaging>

<dependencies>
<dependency>
<groupId>com.rabbitmq</groupId>
<artifactId>amqp-client</artifactId>
<version>3.3.4</version>
</dependency>
<dependency>
<groupId>javax.servlet</groupId>
<artifactId>jstl</artifactId>
<version>1.2</version>
</dependency>
<dependency>
<groupId>javax.servlet</groupId>
<artifactId>javax.servlet-api</artifactId>
<version>3.0.1</version>
<scope>provided</scope>
</dependency>

<!-- This is my worker jar included as dependency -->
<dependency>
<groupId>com.example</groupId>
<artifactId>worker</artifactId>
<version>1.0</version>
</dependency>
</dependencies>
<!-- My local repository containing my worker .jar file-->
<repositories>
<repository>
<id>project.local</id>
<name>project</name>
<url>file:${project.basedir}/localrepo</url>
</repository>
</repositories>


<build>
<plugins>
<plugin>
<groupId>com.heroku.sdk</groupId>
<artifactId>heroku-maven-plugin</artifactId>
<version>1.1.1</version>
</plugin>


<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-war-plugin</artifactId>
<version>2.4</version>
<configuration>
<webResources>
<resource>
<directory>src/web</directory>
</resource>
</webResources>
<archiveClasses>false</archiveClasses>
</configuration>
</plugin>

<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>appassembler-maven-plugin</artifactId>
<version>1.1.1</version>
<configuration>
<assembleDirectory>target</assembleDirectory>
<programs>
<program>
<mainClass>WorkerProcess</mainClass>
<name>worker</name>
</program>
</programs>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>assemble</goal>
</goals>
</execution>
</executions>
</plugin>

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<version>2.3</version>
<executions>
<execution>
<phase>package</phase>
<goals><goal>copy</goal></goals>
<configuration>
<artifactItems>
<artifactItem>
<groupId>com.github.jsimone</groupId>
<artifactId>webapp-runner</artifactId>
<version>8.0.30.2</version>
<destFileName>webapp-runner.jar</destFileName>
</artifactItem>
</artifactItems>
</configuration>
</execution>
</executions>
</plugin>

</plugins>
</build>
</project>

Answer

This is the important message in the logs:

2016-10-31T14:27:43.507545+00:00 app[worker.1]: sh: 0: Can't open target/bin/worker

Does your maven build generate this target/bin/worker script?

I would not expect the CLI deploy to work because it does not include the target dir by default. But you can include it with the --include option.