Node Troubleshooting Guide

Owned by ultimateblockage
Last updated: Jul 18, 2019 by Nathan Bode
4 min read

Expand all   Collapse all

Languages

Node console/log errors

Zend error: no auth mechanism defined

Check that rpcuser and rpcpassword entries exist in the zen.conf file and zen-cli command work properly.

Run the ‘node setup’ which will read the zen.conf file.

Restart the tracker application:  ‘Crtr-c’ and then ‘node app’ or if using pm2: ‘pm2 restart 0’


Too many configuration values changed at once. Reconnect not allowed. Only change one at a time.

Only one of the following may be changed at a time:

  • IP address,

  • node t-address,

  • stake t-address.  


The nodeid must remain the same. The nodeid resides in ~/nodetracker/config/config.json file.

If you have rebuilt or cloned your node and used a new wallet.dat, the node t-address is changed.  Use the same stake address and IP address to connect to the server and authenticate. Once the new node t-address is updated, the stake address can be changed if needed.  


Could not find a registration. Not authenticated.

This usually happens to a registered node when the node has been rebuilt or there have been some node configuration changes or the node configuration is missing the node id.

The node is connecting to the server but the server is not finding the database record.  Check if ~/nodetracker/config/config.json file exist and it has the correct node id.

Node already registered.<node t-address> Authenticated.

The node was previously registered but the nodeid was missing.  The server recognized the node and sent back the nodeid for the node to recreate the file in the config folder.


Not authenticated, Node not found or no nodeid or still initializing.

This happens when the node has not completely authenticated (busy server).  It should correct itself by retrying.


Challenge Issues


Challenge private address balance is 0 at the moment. Cannot perform challenge

"no available balance found or 0"

The challenge did not find a balance in the z-address on the node and cannot send a shielded transaction. Each challenge requires .0002zen (.0001 as an amount and .0001 for a fee). These were found to be the minimums to ensure the shielded transaction is included in the next block with available room.

Z-addresses may temporarily show a 0 balance from when a transaction is sent until the change from the transaction re-appears after being included in a block.  

There are two methods to help ensure there is alway a balance available. With either method only a total of about 0.3 to 05zen is needed on the node. This amount should last a few years based on one challenge a day.  Less if the node fails challenges consistently.

1. Create a second z-address on the node and send a small amount to it. This could be 0.2 in one z-address and 0.2 in the other.  The tracking application will check for an available balance in each address and use the one it finds first. This is the preferred method.

Or

2. Send multiple small amounts (four amounts of 0.1zen) to the single z-address.  This should leave unspent amounts ('notes') in the z-address so a balance check does not fail.




"unable to get blockhash from zen"
"bad-txns-joinsplit-requirements-not-met"


Zen blockchain is most likely not completely synchronized. Wait for it sync.  It can be checked comparing the results of 'zen-cli getblockcount' on the node with the block height at https://explorer.zensystem.io/blocks.



"general exception: std::bad_alloc"


The node has a memory issue or is low on memory and zen can not complete what it is doing. If the node was recently rebooted, make sure the swap file is still available.

A slower system may be able to pass challenges with 5GB, but 6GB leaves a bit more headroom.  Add swap if needed.

Check available memory with 'free -h'.

It should appear similar to this:
              total        used free      shared buff/cache available
Mem:           3.8G 2.9G        187M 1.7M 665M        623M
Swap:          4.0G 975M        3.0G

"exceeded reply time - no txid from node"

A challenge was sent to the node, but either there was no reply from the node with the challenge results or there was no transaction id sent to the tracking server indicating a challenge failure. If there is no entry in the Received cell of the Challenge Results table, then there was no reply for the server to process.

This may also happen if a challenge is interrupted and the server marks the challenge as dead after an hour.

Send another manual challenge using the ‘Send Challenge’ button on the node details page and check the results.

If this issue persists, update zen.conf, adding the following parameter below. Then restart zend and the Node Tracker.

rpcworkqueue=512


'allowed reply variance exceeded'


This is a check between the time the node completes the challenge and the time the server receives the reply from the node.

There may be two trackers running on the node which can result in extended reply times. These may be overwriting each other causing unpredictable results.


If using PM2:


Run 'pm2 list'  There should only be one node processes running.  Stop the second instance.

The process numbers are likely 0 and 1.

  • pm2 stop 1

  • pm2 delete 1

  • pm2 restart 0

  • pm2 save



If only one PM2 process is listed.

  • Check for both PM2 running and the node tracker running manually

  • This may have been started with 'node app.js' or there may be reboot script starting it.

  • If running stop the tracker with Ctrl-c.  Then adjust the configuration for how applications are started for your system.  PM2 can handle this on its own. (See PM2 instructions for the ‘startup’ command on the PM2 web site).


If not using PM2 

Run 'top' to see a list of running processes

In the list of processes, there should only be one where the description includes a reference to 'node' (this is nodejs)

The entry may look like the following. The first number is the PID (Process ID)
3014 nodeadm+  20 0 1220348 39920   1868 S 0.3 4.0 77:00.16 node /home/node

Either kill one of the node processes or it may be simpler to reboot the server.  Once it restarts,
check again for multiple instances of the tracker running.

To stop a process:

  • identify the PID of the process to stop

  • exit top with a 'q'  or 'Ctrl-c'

  • Enter 'kill -9 pid number'  substituting the pid number of the process.

General

Emails not received

This may be from notifications or the buttons on the node detail page for challenges or details.

Check the nodetracker configuration for the proper email and relaunch the tracker.

  • node ~/nodetracker/setup.js
  • sudo systemctl restart zentracker



© 2020 Horizen. All rights reserved.