> For the complete documentation index, see [llms.txt](https://gonka-data-base.gitbook.io/gonka-data-base-en/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://gonka-data-base.gitbook.io/gonka-data-base-en/hosts/node.-testing.md).

# Node. Testing

Tests you can run to check whether the setup is correct.

## General information

It's very frustrating when you seem to have done everything correctly, and still couldn't enter the epoch.

It's best to check everything several times.

{% hint style="info" %}
**Attention!** There may be mistakes in the text because I'm not very familiar with server configuration. More accurately, "very unfamiliar", but these tests helped me.
{% endhint %}

## Check that keys are installed correctly

Sometimes multiple people have configured the nodes or you mixed up the command for assigning the Consensus Public Key, resulting in different keys on the Node and in the Network.

That will definitely prevent you from entering the epoch. Check!

### **Consensus Key check**

#### 🔎 STEP 1. Find the Consensus Public Key on the node

{% hint style="info" %}
**Attention!** \
This check runs on the server where the ML Node is located (or the Network Node — I haven't fully figured it out myself yet, since both nodes are on the same server for me).
{% endhint %}

```
docker exec node wget -qO- "http://127.0.0.1:26657/status" | jq -r '.result.validator_info.pub_key.value'
```

you will get something like:

```
{
  "value": "AD+NQncKPBzqw0u8KcSmlIMqogg7i4nhDfLIgIkGYiY="
}
```

👉 Copy the field `"value"`.**+**

#### 🔎 STEP 2. Find the Consensus Public Key in the network

{% hint style="info" %}
**Attention!** \
This check is done on the server where you created the keys — i.e., not on the Network Node or ML Node.
{% endhint %}

Now see what the network thinks your key is:

> curl -s <http://node2.gonka.ai:8000/chain-api/productscience/inference/inference/participant/**gonka1yplcem8kfe6vm06t4sl8fskm0we2zslxxu90ta>\*\* | jq

{% hint style="info" %}
**Attention!** Replace the bold part with the address of your Hot key.
{% endhint %}

You will get:

```
"AD+NQncKPBzqw0u8KcSmlIMqogg7i4nhDfLIgIkGYiY="
```

As a result you will receive a response like:

> { "participant": { "index": "gonka1yplcem8kfe6vm06t4sl8fskm0we2zslxxu90ta", \
> "address": "gonka1yplcem8kfe6vm06t4sl8fskm0we2zslxxu90ta", \
> "weight": -1, "join\_time": "1771876365572", \
> "join\_height": "2792955", \
> "last\_inference\_time": "0", \
> "inference\_url": "<http://203.168.252.195:8000>", \
> "status": "ACTIVE", \
> "coin\_balance": "0", \
> "validator\_key": "**7GEr4jV5GjCv+C+jKOq3Eh4bwxMVs7kafm7tcWP0EOo=**", \
> "consecutive\_invalid\_inferences": "0", "worker\_public\_key": "", \
> "epochs\_completed": 0, "current\_epoch\_stats": { "inference\_count": "0", "missed\_requests": "0", \
> "earned\_coins": "0",\
> &#x20;"rewarded\_coins": "0", \
> "burned\_coins": "0", \
> "validated\_inferences": "0", \
> "invalidated\_inferences": "0", \
> "invalidLLR": { "value": "0", "exponent": 0 }, \
> "inactiveLLR": { "value": "0", "exponent": 0 },\
> &#x20;"confirmationPoCRatio": null }

We're interested in the value **"validator\_key".**

#### 🔎 STEP 3. Compare them. They must be identical

They should match. Ours do not. \
No wonder we're not getting into the epoch ))

**Reasons** for this mismatch can vary. I think you'll figure out how to fix it yourself.

**How to fix it:** I think you'll manage. It's not complicated.

\---------------------------------------------------------------------------

#### **Find out the model on your node**

```
curl http://localhost:5000/v1/models
```

**Attention!** If nothing is shown, the ML Node might be on a different port. Possible options:

* 5000
* 8000
* 8080
* 9200

i.e., just replace that number in the command.

**Expected response:**

> root\@mlnode-308:/app# curl <http://localhost:5000/v1/models> {"object":"list","data":\[{"id":"**Qwen/Qwen3-235B-A22B-Instruct-2507**-FP8","object":"model","created":1772106402,"owned\_by":"vllm","root":"/root/models/Qwen3-235B-A22B-Instruct-2507-FP8","parent":null,"max\_model\_len":240000,"permission":\[{"id":"modelperm-f9056e19f4b1494c9854c8df9887394b","object":"model\_permission","created":1772106402,"allow\_create\_engine":false,"allow\_sampling":true,"allow\_logprobs":true,"allow\_search\_indices":false,"allow\_view":true,"allow\_fine\_tuning":false,"organization":"\*","group":null,"is\_blocking":false}]}]}root\@mlnode-308:/app#

{% hint style="info" %}
**Attention!** After running this command you enter the Docker container. To continue working with the server command line, exit the container with:\
`exit`\
\
![](https://4037922731-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FvAkz8L4uuD3aggOKw9Xk%2Fuploads%2FONeoSAanJAcIQyaf2YUs%2Fimage.png?alt=media\&token=faa8b636-54dc-4fb1-86f8-8db9f63bdc9f)
{% endhint %}

## Find out the Node configuration

<pre><code>docker exec -it <a data-footnote-ref href="#user-content-fn-1">mlnode-308</a> cat /proc/238/cmdline | tr '\0' ' '
</code></pre>

**Expected response:**

> /usr/bin/python3.12 -m vllm.entrypoints.openai.api\_server --model Qwen/Qwen3-235B-A22B-Instruct-2507-FP8 --dtype float16 --port 5001 --host 0.0.0.0 --max-model-len 240000 --enable-auto-tool-choice --tool-call-parser hermes --tensor-parallel-size 4 --pipeline-parallel-size 2 --enable-expert-parallel --quantization fp8 --gpu-memory-utilization 0.846 --kv-cache-dtype fp8 --swap-space 4 --enforce-eager --cpu-offload-gb 4 --model /root/models/Qwen3-235B-A22B-Instruct-2507-FP8 --served-model-name Qwen/Qwen3-235B-A22B-Instruct-2507-FP8 root\@ecs-99605001-024:\~#

**Attention!** You need to substitute your node name instead of mlnode-308.\
If you forgot it, you can find it with the command:&#x20;

```
docker compose ps
```

**Expected response**- one of these:

<figure><img src="https://4037922731-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FvAkz8L4uuD3aggOKw9Xk%2Fuploads%2Fs6etzPAk2QQN9B7seuJm%2Fimage.png?alt=media&amp;token=16946278-2b44-42aa-8cd1-61dbdaa2406c" alt=""><figcaption></figcaption></figure>

You can list the names of all containers:

```
docker ps --format '{{.Names}}'
```

**Expected response**

<figure><img src="https://4037922731-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FvAkz8L4uuD3aggOKw9Xk%2Fuploads%2FSn4ryCusObPxx84ANuKa%2Fimage.png?alt=media&amp;token=04169d2f-6b0f-43ed-97c8-145b45662d02" alt=""><figcaption></figcaption></figure>

## GPU status

```
curl http://localhost:8080/api/v1/gpu/devices | jq
```

**Expected response:**

<figure><img src="https://4037922731-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FvAkz8L4uuD3aggOKw9Xk%2Fuploads%2FFMCJvXJecZi0jWGgATkC%2Fimage.png?alt=media&amp;token=d9a4203d-5bd3-4939-ad6c-6d8a3634cc7a" alt=""><figcaption></figcaption></figure>

## **Check the declared config**

Check that the node is configured correctly:

```
   
```

<figure><img src="https://4037922731-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FvAkz8L4uuD3aggOKw9Xk%2Fuploads%2FUzUPUntX5X9DoytmXkGq%2Fimage.png?alt=media&amp;token=1261b1c2-c3a3-4bb5-a5d4-cae5581be2d5" alt=""><figcaption></figcaption></figure>

Shows the settings your ML Node is running with. This apparently runs on the ML Node server (if they are separate).

{% hint style="info" %}
**Attention!** These parameters are given as an example. They are certainly outdated. Yours will be different.
{% endhint %}

## Node check with stop

#### Stop the node

<pre><code><strong>curl -sS -X POST "http://127.0.0.1:8080/api/v1/stop" -H "Content-Type: application/json"
</strong>sleep 3
</code></pre>

**Expected response:**

> {"status":"OK"}

#### Check the node's state (status)

```
curl -sS "http://127.0.0.1:8080/api/v1/state"
```

**Expected response:**

> {"state":"STOPPED"}root\@submodel-sxA100-19-14:\~/gonka/deploy/join#

{% hint style="info" %}
If you see something else — run the node stop step again.
{% endhint %}

#### Run the Forced node Test

<pre><code>BLOCK_HEIGHT=$(curl -s "http://node1.gonka.ai:8000/chain-rpc/status" | jq -r '.result.sync_info.latest_block_height')
BLOCK_HASH=$(curl -s "http://node1.gonka.ai:8000/chain-rpc/status" | jq -r '.result.sync_info.latest_block_hash')

curl -sS -X POST "http://127.0.0.1:8080/api/v1/pow/init/generate" \
  -H "Content-Type: application/json" \
  -d "{
  \"node_id\": 0,
  \"node_count\": 1,
  \"block_hash\": \"<a data-footnote-ref href="#user-content-fn-2">D3470A4DDA3D4173BE7C7A55AF52323C19CAC8307DB492CFD004D0C83561068B</a>\",
  \"block_height\": <a data-footnote-ref href="#user-content-fn-3">2753612</a>,
  \"public_key\": \"7GEr4jV5GjCv+C+jK0q3Eh4bwxMVs7kafm7tcWP0E0O=\",
  \"batch_size\": 1,
  \"r_target\": 10.0,
  \"fraud_threshold\": 0.01,
  \"params\": {
    \"dim\": 1792,
    \"n_layers\": 64,
    \"n_heads\": 64,
    \"n_kv_heads\": 64,
    \"vocab_size\": 8196,
    \"ffn_dim_multiplier\": 10.0,
    \"multiple_of\": 8192,
    \"norm_eps\": 1e-5,
    \"rope_theta\": 10000.0,
    \"use_scaled_rope\": false,
    \"seq_len\": 256
  },
  \"url\": \"http://api:9100/v1/poc-batches\"
}"
echo

</code></pre>

**Expected response**

<figure><img src="https://4037922731-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FvAkz8L4uuD3aggOKw9Xk%2Fuploads%2FhNIt74iQkBJ0fBSUTVkV%2Fimage.png?alt=media&amp;token=8e12aa57-0cad-4402-86a7-1e19cb2795e8" alt=""><figcaption></figcaption></figure>

#### Watch the Test progress in the logs

<pre><code>docker logs <a data-footnote-ref href="#user-content-fn-1">mlnode-308</a> --tail 50 -f
</code></pre>

After a few minutes (usually 5–15) the final result should appear. After the PoC finishes:

\
**Expected response:**

<figure><img src="https://4037922731-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FvAkz8L4uuD3aggOKw9Xk%2Fuploads%2Fj55wHAbvait2PjGwBVzN%2Fimage.png?alt=media&amp;token=1fab6351-1b5b-4edf-a7ca-2a942be25d6a" alt=""><figcaption></figcaption></figure>

It's important that CUDA load to 100%

<figure><img src="https://4037922731-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FvAkz8L4uuD3aggOKw9Xk%2Fuploads%2FalGzE8QLkTeWYoOJcHck%2Fimage.png?alt=media&amp;token=6fef0d1e-6edc-4167-8264-5fe9a3978280" alt=""><figcaption></figcaption></figure>

<figure><img src="https://4037922731-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FvAkz8L4uuD3aggOKw9Xk%2Fuploads%2FmtwA0dMh4TZiFsNR9KqM%2Fimage.png?alt=media&amp;token=9b64fcc7-f6a0-4801-8761-abe3ee856c18" alt=""><figcaption></figcaption></figure>

To exit the test press the key combination **CTRL+C**&#x20;

#### Enabling the node

```
curl -sS -X POST "http://127.0.0.1:9200/admin/v1/nodes/node1/enable" -H "Content-Type: application/json"
```

**Expected response**

> {"message":"node enabled successfully","node\_id":"node1"}> \
> root\@submodel-sxA100-19-14:\~/gonka/deploy/join#

#### Check the state of your node:

```
curl http://localhost:8080/api/v1/state
```

**Expected response:**

> root\@ecs-99605001-024:# curl <http://localhost:8080/api/v1/state>> \
> {"state":"INFERENCE"}root\@ecs-99605001-024:#

**Find out the PoV status of your node:**

```
curl http://localhost:8080/api/v1/pow/status
```

**Unexpected response:**

> "detail":"Cannot run POW because MLNode is currently in ServiceState.INFERENCE mode. Please stop ServiceState.INFERENCE first."}root\@ecs-99605001-024:\~#

What the "expected response" should be I don't know yet ))

## Container checks <a href="#id-1.1-proverka-konteinerov" id="id-1.1-proverka-konteinerov"></a>

After startup, first make sure the parameters you set in node-config.json took effect in mlnode

#### Start the container logs **mlnode**

<pre><code>docker logs -f --tail=200 <a data-footnote-ref href="#user-content-fn-4">mlnode-308</a>
</code></pre>

<figure><img src="https://4037922731-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FvAkz8L4uuD3aggOKw9Xk%2Fuploads%2F1ekwDWWve07GMlYoooRg%2Fimage.png?alt=media&amp;token=a411ae07-4158-494f-85b0-2a8a621d8767" alt=""><figcaption></figcaption></figure>

If you see the model loaded like in the screenshot, you can generally exit the container with the key combination **CTRL+C**&#x20;

#### Start the container logs **node**

```
docker logs -f --tail=200 node
```

{% hint style="info" %}
Runs on the Network Node.
{% endhint %}

If the node wasn't synchronized, we should see blockchain "chunks" being fetched

<figure><img src="https://4037922731-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FvAkz8L4uuD3aggOKw9Xk%2Fuploads%2FuTcyYh1QiUM5YxCLv5zD%2Fimage.png?alt=media&amp;token=6637da5d-abca-4852-81c3-e36abf7fbe11" alt=""><figcaption></figcaption></figure>

625 - total count, 160 - last loaded

exit the container with the key combination **CTRL+C**&#x20;

## Check the node's synchronization with the network

```
curl -s "http://127.0.0.1:9200/admin/v1/setup/report" | jq '.checks[] | select(.id == "block_sync")'
```

**Expected response**

<figure><img src="https://4037922731-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FvAkz8L4uuD3aggOKw9Xk%2Fuploads%2FrINTcBGesQtD2Mm6hkBm%2Fimage.png?alt=media&amp;token=a4d6947f-67cf-4415-b3fc-c6f36030e9b7" alt=""><figcaption></figcaption></figure>

So this number should be small. It's the time in seconds since the last block was created.

#### Check the current network block

```
curl -sS http://node1.gonka.ai:8000/chain-rpc/status | jq -r '.result.sync_info.latest_block_height'
```

#### Check the block our node is at

I don't know how yet )

And compare. They should be close.

## Checklist for entering an epoch

Helps understand where to look for the problem.

```
curl -s http://localhost:9200/admin/v1/setup/report | jq '.checks[] | {id,status,message}'
```

Expected response:

<figure><img src="https://4037922731-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FvAkz8L4uuD3aggOKw9Xk%2Fuploads%2FMeFvzFHD5ppJ7r6xLbMB%2Fimage.png?alt=media&amp;token=145b31b3-2333-49d6-a3f7-442a337b5853" alt=""><figcaption></figcaption></figure>

The red arrow marks the check field which **FAIL** is present for absolutely everyone. This parameter **PASS** is only for Gonka master nodes (I think).

The blue arrow points to the field that may be **FAIL** - if you have never entered any epoch yet.

## Check via external address

[\[16.03.2026 21:41\] Anton Bizyaev](https://t.me/gonka_hosts/420):&#x20;

Anton shared an important point when diagnosing a node:

```
curl http://localhost:8080/api/v1/state
```

This command can return `INFERENCE`, and it may seem that everything is working correctly.\
But checking **only localhost is not sufficient**.

Anton says that because of this he lost one epoch:

> At the moment this line cost me one epoch (I'm not blaming anyone — I was not careful enough and spent less time on this than needed).

You must check **the external address**, to ensure the certificate chain works correctly:

```
curl -I https://<FULL_DOMAIN_NAME>:8443/health
```

Expected result:

```
HTTP/2 200 OK
```

If instead you get:

```
Couldn't connect to server
```

— it means the server is unavailable via the external address and the node is not working correctly.

## Links

1. Telegram article: [Gonka - Launch. Or miners in search of profitability. PART\_1](https://telegra.ph/Zagolovok-01-06-13)
2. Telegram article: [Gonka - Launch. Or miners in search of profitability. PART\_2](https://telegra.ph/Gonka---Zapusk-Ili-majnery-v-poiskah-dohodnosti-PART-2-01-08)
3. Gonka Documentation FAQ <https://gonka.ai/FAQ/#how-do-i-fix-errno-validator-signing-info-found-when-starting-from-a-state-sync-snapshot>

END

[^1]: Replace with your node's name.

[^2]: Your data. Collected by an additional command.&#x20;

[^3]: Your data. Collected by an additional command.

[^4]: Replace with your node's Name.
