Troubleshooting

Slack Docker Pulls GitHub edit source

This page is a collection of high-level guides and tips regarding how to diagnose issues encountered in Alluxio.

Note: this doc is not intended to be the full list of Alluxio questions. Join the Alluxio community Slack Channel to chat with users and developers, or post questions on the Alluxio Mailing List.

Where are the Alluxio logs?

Alluxio generates Master, Worker and Client logs under the dir ${ALLUXIO_HOME}/logs. They are named as master.log, master.out, worker.log, worker.out, job_master.log, job_master.out, job_worker.log, job_worker.out and user/user_${USER}.log. Files suffixed with .log are generated by log4j; File suffixed with .out are generated by redirection of stdout and stderr of the corresponding process.

The master and worker logs are useful to understand what the Alluxio Master and Workers are doing, especially when running into any issues. If you do not understand the error messages, search for them in the Mailing List, in the case the problem has been discussed before. You can also join our Slack channel and seek help there.

The client-side logs are also helpful when Alluxio service is running but the client cannot connect to the servers. Alluxio client emits logging messages through log4j, so the location of the logs is determined by the client side log4j configuration used by the application.

For more information about logging, please check out this page.

Alluxio remote debug

Java remote debugging makes it easier to debug Alluxio at the source level without modifying any code. You will need to append the JVM remote debugging parameters and start a debugging server. There are several ways to append the remote debugging parameters; you can export the following configuration properties in shell or alluxio-env.sh:

  1. export ALLUXIO_WORKER_JAVA_OPTS="$ALLUXIO_JAVA_OPTS -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=6606"
  2. export ALLUXIO_MASTER_JAVA_OPTS="$ALLUXIO_JAVA_OPTS -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=6607"
  3. export ALLUXIO_USER_DEBUG_JAVA_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=6609"

If you want to debug shell commands, you can add the -debug flag to start a debug server with the JVM debug parameters ALLUXIO_USER_DEBUG_JAVA_OPTS, such as alluxio fs -debug ls /.

suspend = y/n will decide whether the JVM process wait until the debugger connects. If you want to debug with the shell command, set the suspend = y. Otherwise, you can set suspend = n to avoid unnecessary waiting time.

After starting the master or worker, use Eclipse, IntelliJ IDEA, or another java IDE, create a new java remote configuration, set the debug server’s host and port, and start the debug session. If you set a breakpoint which can be reached, the IDE will enter debug mode and you can inspect the current context’s variables, call stack, thread list, and expression evaluation.

Alluxio collectInfo command

Alluxio has a collectInfo command that collect information to troubleshoot an Alluxio cluster. collectInfo will run a set of sub-commands that each collects one aspect of system information, as explained below. In the end the collected information will be bundled into one tarball which contains a lot of information regarding your Alluxio cluster. The tarball size mostly depends on your cluster size and how much information you are collecting. For example, collectLog operation can be costly if you have huge amounts of logs. Other commands typically do not generate files larger than 1MB. The tarball will help you troubleshoot your cluster. Or you can shared the tarball with someone you trust to help troubleshoot your Alluxio cluster.

The collectInfo command will SSH to each node and execute the set of sub-commands. In the end of execution the collected information will be written to files and tarballed. Each individual tarball will be collected to the issuing node. Then all the tarballs will be bundled into the final tarball, which contains all information about the Alluxio cluster.

NOTE: Be careful if your configuration contains credentials like AWS keys! You should ALWAYS CHECK what is in the tarball and REMOVE the sensitive information from the tarball before sharing it with someone!

Collect Alluxio cluster information

collectAlluxioInfo will run a set of Alluxio commands that collect information about the Alluxio cluster, like bin/alluxio fsadmin report etc. When the Alluxio cluster is not running, this command will fail to collect some information.

NOTE: The configuration parameters will be collected with alluxio getConf --master, which obfuscates the credential fields passed to Alluxio as properties.

Collect Alluxio configuration files

collectConfig will collect all the configuration files under ${alluxio.work.dir}/conf.

WARNING: If you put credential fields in the configuration files, DO NOT share the collected tarball with anybody unless you have manually obfuscated them in the tarball!

Collect Alluxio logs

collectLog will collect all the logs under ${alluxio.work.dir}/logs.

NOTE: Roughly estimate how much log you are collecting before executing this command!

Collect Alluxio metrics

collectMetrics will collect Alluxio metrics served at http://${alluxio.master.hostname}:${alluxio.master.web.port}/metrics/json/ by default. The metrics will be collected multiple times to see the progress.

Collect JVM information

collectJvmInfo will collect information about the existing JVMs on each node. This is done by running a jps command then jstack on each found JVM process. This will be done multiple times to see if the JVMs are making progress.

Collect system information

collectEnv will run a set of bash commands to collect information about the running node. This runs system troubleshooting commands like env, hostname, top, ps etc.

WARNING: If you stored credential fields in environment variables like AWS_ACCESS_KEY or in process start parameters like -Daws.access.key=XXX, DO NOT share the collected tarball with anybody unless you have manually obfuscated them in the tarball!

Collect all information mentioned above

all will run all the sub-commands above.

Command options

The collectInfo command has the below options.

  1. $ bin/alluxio collectInfo 
  2.     [--max-threads <threadNum>] 
  3.     [--local] 
  4.     [--help]
  5.     [--additional-logs <filename-prefixes>] 
  6.     [--exclude-logs <filename-prefixes>] 
  7.     [--include-logs <filename-prefixes>] 
  8.     [--start-time <datetime>] 
  9.     [--end-time <datetime>]
  10.     COMMAND <outputPath>

<outputPath> is the directory you want the final tarball to be written into.

Options:

  1. --max-threads threadNum option configures how many threads to use for concurrently collecting information and transmitting tarballs. When the cluster has a large number of nodes, or large log files, the network IO for transmitting tarballs can be significant. Use this parameter to constrain the resource usage of this command.

  2. --local option specifies the collectInfo command to run only on localhost. That means the command will only collect information about the localhost.

  3. --help option asks the command to print the help message and exit.

  4. --additional-logs <filename-prefixes> specifies extra log file name prefixes to include. By default, only log files recognized by Alluxio will be collected by the collectInfo command. The recognized files include below:

    1. logs/master.log*, 
    2. logs/master.out*, 
    3. logs/job_master.log*, 
    4. logs/job_master.out*, 
    5. logs/master_audit.log*, 
    6. logs/worker.log*, 
    7. logs/worker.out*, 
    8. logs/job_worker.log*, 
    9. logs/job_worker.out*, 
    10. logs/proxy.log*, 
    11. logs/proxy.out*, 
    12. logs/task.log*, 
    13. logs/task.out*, 
    14. logs/user/*

    Other than mentioned above, --additional-logs <filename-prefixes> specifies that files whose names start with the prefixes in <filename-prefixes> should be collected. This will be checked after the exclusions defined in --exclude-logs. <filename-prefixes> specifies the filename prefixes, separated by commas.

  5. --exclude-logs <filename-prefixes> specifies file name prefixes to ignore from the default list.

  6. --include-logs <filename-prefixes> specifies only to collect files whose names start with the specified prefixes, and ignore all the rest. You CANNOT use --include-logs option together with either --additional-logs or --exclude-logs, because it is ambiguous what you want to include.

  7. --end-time <datetime> specifies a datetime after which the log files can be ignored. A log file will be ignore if the file was created after this end time. The first couple of lines of the log file will be parsed, in order to infer when the log file started. The <datetime> is a datetime string like 2020-06-27T11:58:53. The parsable datetime formats include below:

    1. "2020-01-03 12:10:11,874"
    2. "2020-01-03 12:10:11"
    3. "2020-01-03 12:10"
    4. "20/01/03 12:10:11"
    5. "20/01/03 12:10"
    6. 2020-01-03T12:10:11.874+0800
    7. 2020-01-03T12:10:11
    8. 2020-01-03T12:10

  8. --start-time <datetime> specifies a datetime before with the log files can be ignored. A log file will be ignored if the last modified time is before this start time.

Setup FAQ

Q: I’m new to Alluxio and cannot set up Alluxio on my local machine. What should I do?

A: Check ${ALLUXIO_HOME}/logs to see if there are any master or worker logs. Look for any errors in these logs. Double check if you missed any configuration steps in Running-Alluxio-Locally.

Typical issues:

  • ALLUXIO_MASTER_MOUNT_TABLE_ROOT_UFS is not configured correctly.
  • If running ssh localhost fails, make sure the public SSH key for the host is added in ~/.ssh/authorized_keys.

Q: I’m trying to deploy Alluxio in a cluster with Spark and HDFS. Are there any suggestions?

A: Please follow Running-Alluxio-on-a-Cluster, Configuring-Alluxio-with-HDFS.

Tips:

  • The best performance gains occur when Alluxio workers are co-located with the nodes of the computation frameworks.
  • You can use Mesos and Yarn integration if you are already using Mesos or Yarn to manage your cluster.
  • If the under storage is remote (like S3 or remote HDFS), using Alluxio can be especially beneficial.

Q: Why do I see “Unsupported major.minor version 52.0” error when I start Alluxio?

A: Alluxio requires Java 8 runtime to function properly. If this error is seen at the start of Alluxio master or worker, please setup your environment so that the default Java version is 8. If you see this error while using an application to access files on Alluxio, please make sure your application is running on Java 8.

Usage FAQ

Q: Why do I see exceptions like “No FileSystem for scheme: alluxio”?

A: This error message is seen when your applications (e.g., MapReduce, Spark) try to access Alluxio as an HDFS-compatible file system, but the alluxio:// scheme is not recognized by the application. Please make sure your HDFS configuration file core-site.xml (in your default hadoop installation or spark/conf/ if you customize this file for Spark) has the following property:

  1. <configuration>
  2.   <property>
  3.     <name>fs.alluxio.impl</name>
  4.     <value>alluxio.hadoop.FileSystem</value>
  5.   </property>
  6. </configuration>

Q: Why do I see exceptions like “java.lang.RuntimeException: java.lang.ClassNotFoundException: Class alluxio.hadoop.FileSystem not found”?

A: This error message is seen when your applications (e.g., MapReduce, Spark) try to access Alluxio as an HDFS-compatible file system, the alluxio:// scheme has been configured correctly, but the Alluxio client jar is not found on the classpath of your application. Depending on the computation frameworks, users usually need to add the Alluxio client jar to their class path of the framework through environment variables or properties on all nodes running this framework. Here are some examples:

  • For MapReduce jobs, you can append the client jar to $HADOOP_CLASSPATH:
  1. $ export HADOOP_CLASSPATH=/<PATH_TO_ALLUXIO>/client/alluxio-2.3.0-client.jar:${HADOOP_CLASSPATH}
  • For Spark jobs, you can append the client jar to $SPARK_CLASSPATH:
  1. $ export SPARK_CLASSPATH=/<PATH_TO_ALLUXIO>/client/alluxio-2.3.0-client.jar:${SPARK_CLASSPATH}

Alternatively, add the following lines to spark/conf/spark-defaults.conf:

  1. spark.driver.extraClassPath /<PATH_TO_ALLUXIO>/client/alluxio-2.3.0-client.jar
  2. spark.executor.extraClassPath /<PATH_TO_ALLUXIO>/client/alluxio-2.3.0-client.jar

  • For Presto, put Alluxio client jar /<PATH_TO_ALLUXIO>/client/alluxio-2.3.0-client.jar into the directory ${PRESTO_HOME}/plugin/hive-hadoop2/ Since Presto has long running processes, ensure they are restarted after the jar has been added.

  • For Hive, set HIVE_AUX_JARS_PATH in conf/hive-env.sh:

  1. $ export HIVE_AUX_JARS_PATH=/<PATH_TO_ALLUXIO>/client/alluxio-2.3.0-client.jar:${HIVE_AUX_JARS_PATH}

Since Hive has long running processes, ensure they are restarted after the jar has been added.

If the corresponding classpath has been set but exceptions still exist, users can check whether the path is valid by:

  1. $ ls /<PATH_TO_ALLUXIO>/client/alluxio-2.3.0-client.jar

Q: I’m seeing error messages like “Frame size (67108864) larger than max length (16777216)”. What is wrong?

A: This problem can be caused by different possible reasons.

  • Please double check if the port of Alluxio master address is correct. The default listening port for Alluxio master is port 19998, while a common mistake causing this error message is due to using a wrong port in master address (e.g., using port 19999 which is the default Web UI port for Alluxio master).
  • Please ensure that the security settings of Alluxio client and master are consistent. Alluxio provides different approaches to authenticate users by configuring alluxio.security.authentication.type. This error happens if this property is configured with different values across servers and clients (e.g., one uses the default value NOSASL while the other is customized to SIMPLE). Please read Configuration-Settings for how to customize Alluxio clusters and applications.

Q: I’m copying or writing data to Alluxio while seeing error messages like “Failed to cache: Not enough space to store block on worker”. Why?

A: This error indicates insufficient space left on Alluxio workers to complete your write request.

  • Check if you have any files unnecessarily pinned in memory and unpin them to release space. See Command-Line-Interface for more details.
  • Increase the capacity of workers by changing alluxio.worker.memory.size property. See Configuration for more description.

Q: I’m writing a new file/directory to Alluxio and seeing journal errors in my application

A: When you see errors like “Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try”, it is because Alluxio master failed to update journal files stored in a HDFS directory according to the property alluxio.master.journal.folder setting. There can be multiple reasons for this type of errors, typically because some HDFS datanodes serving the journal files are under heavy load or running out of disk space. Please ensure the HDFS deployment is connected and healthy for Alluxio to store journals when the journal directory is set to be in HDFS.

Q: I added some files in under file system. How can I reveal the files in Alluxio?

By default, Alluxio loads the list of files the first time a directory is visited. Alluxio will keep using the cached file list regardless of the changes in the under file system. To reveal new files from under file system, you can use the command alluxio fs ls -R -Dalluxio.user.file.metadata.sync.interval=${SOME_INTERVAL} /path or by setting the same configuration property in masters’ alluxio-site.properties. The value for the configuration property is used to determine the minimum interval between two syncs. You can read more about loading files from underfile system here.

Q: I see an error “Block ?????? is unavailable in both Alluxio and UFS” while reading some file. Where is my file?

A: When writing files to Alluxio, one of the several write type can be used to tell Alluxio worker how the data should be stored:

MUST_CACHE: data will be stored in Alluxio only

CACHE_THROUGH: data will be cached in Alluxio as well as written to UFS

THROUGH: data will be only written to UFS

ASYNC_THROUGH: data will be stored in Alluxio synchronously and then written to UFS asynchronously

By default the write type used by Alluxio client is ASYNC_THROUGH, therefore a new file written to Alluxio is only stored in Alluxio worker storage, and can be lost if a worker crashes. To make sure data is persisted, either use CACHE_THROUGH or THROUGH write type, or increase alluxio.user.file.replication.durable to an acceptable degree of redundancy.

Another possible cause for this error is that the block exists in the file system, but no worker has connected to master. In that case the error will go away once at least one worker containing this block is connected.

Q: I’m running an Alluxio shell command and it hangs without giving any output. What’s going on?

A: Most Alluxio shell commands require connecting to Alluxio master to execute. If the command fails to connect to master it will keep retrying several times, appearing as “hanging” for a long time. It is also possible that some command can take a long time to execute, such as persisting a large file on a slow UFS. If you want to know what happens under the hood, check the user log (stored as ${ALLUXIO_HOME}/logs/user_${USER_NAME}.log by default) or master log (stored as ${ALLUXIO_HOME}/logs/master.log on the master node by default).

Performance FAQ

Q: I tested Alluxio/Spark against HDFS/Spark (running simple word count of GBs of files). Why is there no discernible performance difference?

A: Alluxio accelerates your system performance by leveraging temporal or spatial locality using distributed in-memory storage (and tiered storage). If your workloads don’t have any locality, you will not see noticeable performance boost.

For a comprehensive guide on tuning performance of Alluxio cluster, please check out this page.

Environment

Alluxio can be configured under a variety of modes, in different production environments. Please make sure the Alluxio version being deployed is update-to-date and supported.

When posting questions on the Mailing List or Slack channel, please attach the full environment information, including

  • Alluxio version
  • OS version
  • Java version
  • UnderFileSystem type and version
  • Computing framework type and version
  • Cluster information, e.g. the number of nodes, memory size in each node, intra-datacenter or cross-datacenter