> ## Documentation Index
> Fetch the complete documentation index at: https://docs.monad.xyz/llms.txt
> Use this file to discover all available pages before exploring further.

# Full Replay

Full Replay allows a node to catch up on missed blocks by simply retrieving and replaying them
serially. This is helpful for nodes that want all intermediary state and transactional artifacts
to be generated. For example, an RPC provider would probably want this to ensure that they can
respond to requests like `eth_call`, `eth_getBalance`, or `eth_estimateGas` for blocks
that were skipped.

## Context

A node that has not locally executed block `forkpoint.root - delay` will statesync on startup. Also, nodes don’t serve blocksync requests more than statesync\_threshold (600) blocks ago. Statesync is the only in-protocol way for that node to recover.

This document describes an *alternate* way to recover a node while backfilling historical state in the event that a node has been down for longer than the blocksync provision window (600 blocks).

Note that if `statesync_threshold` in `node.toml` is set to a value larger than that, blocksync will fail.

<Warning title="Access to a healthy node required">
  In order to recover the faulty node with complete historical state, the below
  procedure *requires SSH access to a node that was healthy* throughout the faulty node's downtime.

  Let `REMOTE_HOST` be the healthy node that can be used for recovery.
</Warning>

## Procedure

1. SSH into the faulty node as `monad` user.
2. Ensure `statesync_threshold = 600` in `node.toml`

```bash theme={null}
$ grep statesync_threshold /home/monad/monad-bft/config/node.toml
statesync_threshold = 600
```

1. Stop the monad services

```bash theme={null}
sudo systemctl stop monad-bft monad-execution monad-rpc
```

1. Run this script to copy missing blocks, then run execution up to that point. You will need to run it several times since the tip of the chain will continue advancing while this script is executing.

   1. NOTE: You will need to manually interrupt the process (Ctrl - C) once output stops.

   2. Copy this script and name it `manual-sync.sh`

      ```bash theme={null}
      #!/usr/bin/env bash
      set -euo pipefail

      # Load config first so SSH_PORT, REMOTE_HOST, CHAIN etc are set
      source .env

      : "${SSH_PORT:?SSH_PORT must be set}"
      : "${REMOTE_HOST:?REMOTE_HOST must be set}"
      : "${CHAIN:?CHAIN must be set}"

      echo "Reading files from $REMOTE_HOST"
      echo "Using ssh port: $SSH_PORT"

      # Copy proposed and finalized header symlinks first
      rsync -avP -e "ssh -p $SSH_PORT" \
        "monad@$REMOTE_HOST:/home/monad/monad-bft/ledger/headers/*_head" \
        /home/monad/monad-bft/ledger/headers/

      # Copy new headers and bodies, but don't overwrite existing ones
      rsync -avP --ignore-existing -e "ssh -p $SSH_PORT" \
        "monad@$REMOTE_HOST:/home/monad/monad-bft/ledger/" \
        /home/monad/monad-bft/ledger/

      # Note: the --state-sync flag is NOT present
      /usr/local/bin/monad \
        --chain "$CHAIN" \
        --db /dev/triedb \
        --block_db /home/monad/monad-bft/ledger \
        --sq_thread_cpu 1 \
        --log_level INFO
      ```

   3. Run the script
      ```bash theme={null}
      REMOTE_HOST=node1.<provider>.com SSH_PORT=22 CHAIN=monad_mainnet bash manual-sync-step-1.sh
      ```

2. Once the first script completes in under 1 min, run this script:

   1. Copy this and name it `manual-sync-step-2.sh`
      ```bash theme={null}
      #!/usr/bin/env bash
      set -euo pipefail

      # Load config first so SSH_PORT, REMOTE_HOST, CHAIN etc are set
      source .env

      : "${SSH_PORT:?SSH_PORT must be set}"
      : "${REMOTE_HOST:?REMOTE_HOST must be set}"
      : "${CHAIN:?CHAIN must be set}"

      # Copy current forkpoint
      rsync -av -e "ssh -p $SSH_PORT" \
        "monad@$REMOTE_HOST:/home/monad/monad-bft/config/forkpoint/forkpoint.rlp" \
        /home/monad/monad-bft/config/forkpoint/
      rsync -av -e "ssh -p $SSH_PORT" \
        "monad@$REMOTE_HOST:/home/monad/monad-bft/config/forkpoint/forkpoint.toml" \
        /home/monad/monad-bft/config/forkpoint/forkpoint.toml
      rsync -av -e "ssh -p $SSH_PORT" \
        "monad@$REMOTE_HOST:/home/monad/monad-bft/config/validators/validators.toml" \
        /home/monad/monad-bft/config/validators/validators.toml

      systemctl start monad-bft monad-execution monad-rpc
      ```
   2. Run the script
      ```bash theme={null}
      REMOTE_HOST=node1.<provider>.com SSH_PORT=22 CHAIN=monad_mainnet bash manual-sync-step-2.sh
      ```

## Check

To check that statesync has been avoided, send an `eth_getBlockByNumber` RPC request
for blocks finalized while the node was down.

```bash theme={null}
curl http://localhost:8080 \
  -X POST \
  -H "Content-Type: application/json" \
  --data '{"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["0xdedbeef", false],"id":1}'
```