Skip to main content

Running an Archive Server

Please see The Data Waterfall for an overview of the different sources of archive data. This page is dedicated to operational details on running an Archive Server.

For anyone looking to reliably serve historical transactional data, it is recommended to run multiple Archive Servers, fed by multiple full nodes running the Archive Writer process.

Suggested configuration:

  • 2 Archive Servers running MongoDB + monad-indexer
  • 2 Archive Writer nodes, aka full node + monad-archiver
  • Many full nodes serving RPC requests, connected to both Archive Servers for historical transactional data
  • CPU: 16 cores
  • RAM: minimum 64GB; prefer >512GB for best performance
  • Storage: minimum recommended 16TB; preferred 32TB+
    • NVMe SSD
    • RAID 10
    • Minimum 10K IOPS sustained
  • Network: >1GbE, scale higher as needed when serving more RPC servers

Fresh installation​

These instructions assume an existing full node (for Archive Writer) and a new host (for Archive Server).

On your Archive Server:​

  1. Add the following section to /home/monad/.env

    # Replace <db username> and <db pwd> with your actual values
    DB_USER="<your db username>"
    DB_PWD="<your db pwd>"
    DB_VOLUME="~/archive-db-data"
    

    # Note: source and sink should generally be the same local MongoDB.
    # Indexer reads from:
    BLOCK_DATA_SOURCE="mongodb mongodb://${DB_USER}:${DB_PWD}@0.0.0.0:27017 archive-db"
    # Indexer writes to:
    ARCHIVE_SINK="mongodb mongodb://${DB_USER}:${DB_PWD}@0.0.0.0:27017 archive-db"
    OTEL_ENDPOINT="http://0.0.0.0:4317"

  2. Run mongodb. Instructions show how to run in Docker, but operators can run however they choose.

    archive-db:
      image: mongo:latest
      command: mongod --bind_ip 0.0.0.0
      networks:
        - host
      environment:
        MONGO_INITDB_ROOT_USERNAME: ${DB_USER}
        MONGO_INITDB_ROOT_PASSWORD: ${DB_PWD}
      ports:
        - "27017:27017"
      volumes:
        - ${DB_VOLUME}:/data/db
      logging:
        driver: journald
        options:
          tag: "mongo"

  3. Start the db

    # create directory from .env db volume
    source /home/monad/.env
    sudo mkdir -p $DB_VOLUME
    sudo chown -R monad:monad $DB_VOLUME
    sudo chmod 700 $DB_VOLUME
    

    docker compose up archive-db -d

On your Archive Writer host:​

  1. Add the following section to /home/monad/.env

    ###############################################################
    ################### Vars that do not change ###################
    ###############################################################
    # Replace with your Archive Server credentials
    ARCHIVE_DB_USER="<your db username>"
    ARCHIVE_DB_PWD="<your db pwd>"
    ARCHIVE_DB_HOST="<your db hostname>"
    

    ## Where block data gets written (your Archive Server)
    ARCHIVE_SINK="mongodb mongodb://${ARCHIVE_DB_USER}:${ARCHIVE_DB_PWD}@${ARCHIVE_DB_HOST}:27017 archive-db"
    ## Change this if your ledger folder is in a different location
    BFT_BLOCK_PATH=~/monad-bft/ledger
    OTEL_ENDPOINT=http://0.0.0.0:4317
    ## This can be significantly higher during backfill
    MAX_CONCURRENT_BLOCKS=50
    

    

    ###############################################################
    ####### Backfill from Genesis (initial configuration) #########
    ###############################################################
    BLOCK_DATA_SOURCE="aws testnet-ltu-032-0 50" # testnet
    BLOCK_DATA_SOURCE="aws mainnet-deu-010-0 50" # mainnet
    ## Alternatively you can use your other ArchiveDB if it has already been backfilled
    ## If using another Archive Server, define variables for it and use them:
    #OTHER_DB_USER="<other db username>"
    #OTHER_DB_PWD="<other db pwd>"
    #OTHER_DB_HOST="<other db hostname>"
    #BLOCK_DATA_SOURCE="mongodb mongodb://${OTHER_DB_USER}:${OTHER_DB_PWD}@${OTHER_DB_HOST}:27017 archive-db"
    FALLBACK_BLOCK_DATA_SOURCE="aws testnet-ltu-032-0 50" # testnet
    FALLBACK_BLOCK_DATA_SOURCE="aws mainnet-deu-010-0 50" # mainnet
    

    ###############################################################
    ###################### Normal Operation #######################
    ###############################################################
    ## Archiver checks triedb first for block data
    BLOCK_DATA_SOURCE="triedb /dev/triedb 5000"
    

    ### FALLBACK_BLOCK_DATA_SOURCE ###
    ## This is used whenever data is missing from BLOCK_DATA_SOURCE
    ## Normally used after state-sync
    ## If you have another Archive Server for redundancy, configure it here:
    #OTHER_DB_USER="<other db username>"
    #OTHER_DB_PWD="<other db pwd>"
    #OTHER_DB_HOST="<other db hostname>"
    #FALLBACK_BLOCK_DATA_SOURCE="mongodb mongodb://${OTHER_DB_USER}:${OTHER_DB_PWD}@${OTHER_DB_HOST}:27017 archive-db"
    

    ## Alternatively you can use category-labs aws bucket
    ## Note: YOU pay for S3 egress costs if pulling from this bucket
    FALLBACK_BLOCK_DATA_SOURCE="aws testnet-ltu-032-0 50" # testnet
    FALLBACK_BLOCK_DATA_SOURCE="aws mainnet-deu-010-0 50" # mainnet

  2. Backfilling:

    note

    As of v0.12.3, --start-block is no longer supported as a daemon argument. Instead, use the set-start-block subcommand to set the starting block marker imperatively before starting the daemon. This prevents accidental progress loss from systemd overrides.

    # [Optional] Set a specific start block if needed
    # This is typically only needed for initial setup or recovery scenarios
    # Example: start archiving from block 1000000
    monad-archiver set-start-block --block 1000000 --archive-sink "${ARCHIVE_SINK}"
    

    # For async backfill marker, add the --async-backfill flag:
    # monad-archiver set-start-block --block 1000000 --archive-sink "${ARCHIVE_SINK}" --async-backfill
    

    # Start the archiver daemon
    sudo systemctl start monad-archiver
    # Should show it running
    # Double check the arguments look correct based on the above ^
    systemctl status monad-archiver
    # Expect many:
    # > INFO: Successfully archived block block_num=X
    journalctl -u monad-archiver -o cat

  3. Once monad-archiver has caught up to the chain tip you will see:

    INFO: Nothing to process

  4. This means backfilling is done and you should

    1. Comment out BACKFILL section in your .env and uncomment Normal Operation
    2. Restart: systemctl restart monad-archiver
    3. Verify status and logs as above

On your ArchiveDB host​

  1. Start monad-indexer

    sudo systemctl start monad-indexer
    systemctl status monad-indexer
    # Expect many:
    # > INFO: Indexing block...
    # > INFO: Index spot-check successful
    journalctl -u monad-indexer -o cat

[Optional] Set up monad-archive-checker​

Typically only 1 checker is run per organization, but more can be run if desired

  1. Add the following systemd override to monad-archive-checker

    sudo systemctl edit monad-archive-checker
    [Service]
    ExecStart=
    ExecStart=/usr/local/bin/monad-archive-checker \
        # Example --bucket my-org-mainnet-checker
        # Note: storing checker state in local mongo or filesystem
        #       will be supported in a future release
        --bucket <S3 Bucket for storing checker state>
        checker
        # Example of comparing a local self-hosted mongo against a Category Labs aws bucket
        # --init-replicas "<aws mainnet-deu-009-0 50,mongodb mongodb://<username>:<pwd>@<db hostname>:27017 archive-db>,mongodb mongodb://<username>:<pwd>@<second db hostname>:27017 archive-db>"
        --init-replicas "<ArchiveDB 1>,<ArchiveDB 2>"

  2. Reload and Restart

    sudo systemctl daemon-reload
    sudo systemctl restart monad-archive-checker
    systemctl status monad-archive-checker
    

    # Check for errors
    journalctl -u monad-archive-checker -o cat -f