Fetching large responses with Chainlink AnyAPI

Storing large amounts of text- or binary data on a blockchain is normally ill-advised. Blockspace is expensive and languages like Solidity don't really have fully-fleshed out string-manipulation libraries. Nonethless sometimes you need to fetch a string.

Oracles and Operators

Fetching strings or bytes using Chainlink's AnyAPI approach requires your node operator to use the newer Operator.sol which can be deployed from Remix using the following contract

// SPDX-License-Identifier: MIT
pragma solidity ^0.7.0;
import "@chainlink/contracts/src/v0.7/Operator.sol";

The constructor takes two parameters LINK and OWNER. LINK needs to be set to the Link token address for the network that contract is being deployed to. The OWNER parameter is the wallet address of Operator owner/deployer and NOT the node operator address (as is the case when deploying the older oracle.sol contract)

The next step, once the Operator contract is deployed, is to associate the Operator contract with the Chainlink node address. To do so using Remix call the setAuthorizedSenders() function and specify the node address.

// specify the node address
setAuthorizedSenders(["0x67b65DDb04282668Cc2Cb85e9381Da0E249B7258"])

Get > Bytes

The node operator needs deploy a job similar to the one below. Pay special attention to the two contract addresses references, explained next.

type = "directrequest"
schemaVersion = 1
name = "Get > Bytes"
maxTaskDuration = "0s"
contractAddress = "0x1314E350Fc5a3896E2d66C43A83D9391E914a004"
minIncomingConfirmations = 0
observationSource = """
    decode_log   [type="ethabidecodelog"
                  abi="OracleRequest(bytes32 indexed specId, address requester, bytes32 requestId, uint256 payment, address callbackAddr, bytes4 callbackFunctionId, uint256 cancelExpiration, uint256 dataVersion, bytes data)"
                  data="$(jobRun.logData)"
                  topics="$(jobRun.logTopics)"]

    decode_cbor  [type="cborparse" data="$(decode_log.data)"]
    fetch        [type="http" method=GET url="$(decode_cbor.get)"]
    parse        [type="jsonparse" path="$(decode_cbor.path)" data="$(fetch)"]
    encode_large [type="ethabiencode"
                abi="(bytes32 requestId, bytes _data)"
                data="{\\"requestId\\": $(decode_log.requestId), \\"_data\\": $(parse)}"
                ]
    encode_tx  [type="ethabiencode"
                abi="fulfillOracleRequest2(bytes32 requestId, uint256 payment, address callbackAddress, bytes4 callbackFunctionId, uint256 expiration, bytes calldata data)"
                data="{\\"requestId\\": $(decode_log.requestId), \\"payment\\":   $(decode_log.payment), \\"callbackAddress\\": $(decode_log.callbackAddr), \\"callbackFunctionId\\": $(decode_log.callbackFunctionId), \\"expiration\\": $(decode_log.cancelExpiration), \\"data\\": $(encode_large)}"
                ]

    submit_tx    [type="ethtx" to="0x1314E350Fc5a3896E2d66C43A83D9391E914a004" data="$(encode_tx)"]

    decode_log -> decode_cbor -> fetch -> parse  -> encode_large -> encode_tx -> submit_tx
"""

The two contract addresses (both 0x1314..004 above) need to point to the deployed Operator contract.

Fetching a large response

Finally to call the job on the node via the operator contract follow the example below (which is the example from the main docs).

//SPDX-License-Identifier: MIT
pragma solidity ^0.8.7;

import "@chainlink/contracts/src/v0.8/ChainlinkClient.sol";

/**
 * Request testnet LINK and ETH here: https://faucets.chain.link/
 * Find information on LINK Token Contracts and get the latest ETH and LINK faucets here: https://docs.chain.link/docs/link-token-contracts/
 */

/**
 * @notice DO NOT USE THIS CODE IN PRODUCTION. This is an example contract.
 */
contract GenericLargeResponse is ChainlinkClient {
  using Chainlink for Chainlink.Request;

  // variable bytes returned in a single oracle response
  bytes public data;
  string public image_url;

  /**
   * @notice Initialize the link token and target oracle
   * @dev The oracle address must be an Operator contract for multiword response
   *
   *
   */
  constructor(
  ) {
    setChainlinkToken(0x326C977E6efc84E512bB9C30f76E30c160eD06FB);
    setChainlinkOracle(0x1314E350Fc5a3896E2d66C43A83D9391E914a004);
  }

  /**
   * @notice Request variable bytes from the oracle
   */
  function requestBytes(
  )
    public
  {
    bytes32 specId = "490d815cbbb74a0db1d17e7aae3deb84";
    uint256 payment = 100000000000000000;
    Chainlink.Request memory req = buildChainlinkRequest(specId, address(this), this.fulfillBytes.selector);
    req.add("get","https://ipfs.io/ipfs/QmZgsvrA1o1C8BGCrx6mHTqR1Ui1XqbCrtbMVrRLHtuPVD?filename=big-api-response.json");
    req.add("path", "image");
    sendOperatorRequest(req, payment);
  }

  event RequestFulfilled(
    bytes32 indexed requestId,
    bytes indexed data
  );

  /**
   * @notice Fulfillment function for variable bytes
   * @dev This is called by the oracle. recordChainlinkFulfillment must be used.
   */
  function fulfillBytes(
    bytes32 requestId,
    bytes memory bytesData
  )
    public
    recordChainlinkFulfillment(requestId)
  {
    emit RequestFulfilled(requestId, bytesData);
    data = bytesData;
    image_url = string(data);
  }

}

Important items to note that differ from the canonical Get > Uint256 example are the use of

    setChainlinkOracle(0x1314E350Fc5a3896E2d66C43A83D9391E914a004);

in the constructor which points to the Operator contract address.

The specId in requestBytes() refers to the node's jobId.

    bytes32 specId = "490d815cbbb74a0db1d17e7aae3deb84";

Finally note how Operator.sol requires us to send requests with sendChainlinkRequestTo() in requestBytes():

    sendOperatorRequest(req, ORACLE_PAYMENT);