0

I tried to scrape the coinmarketcap dot com with selenium webdriver, but found an error of list index of range and did not understand why this happened.

driver.get('https://coinmarketcap.com/')

#stored the all rows of coins (that is 100 on first page)
all_tr = driver.find_elements(By.XPATH,'//tbody/tr')

#scrape coin name and coin url
for index, i in enumerate(all_tr):
    coin_name = driver.find_elements(By.XPATH,'//td[3]/div/a/div/div/p')[index].text
    coin_url = driver.find_elements(By.XPATH,'//td[3]/div/a')[index].get_attribute('href')
    print(coin_name,coin_url)

It goes fine till 18 results and then generates an error of list index out of range.

enter image description here

Improve this question

1 Answer 1

Reset to default
1

The reason that happens is because the elements don't become visible until you scroll down on chrome. You can use a script that will scroll down and up after every loop in order to keep the elements visible. I luckily have one that I use for similar scenarios, here is what your code will look like after:

def make_elements_visible():
    last_height = driver.execute_script("return document.body.scrollHeight")
    while True:
        driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
        new_height =driver.execute_script("return document.body.scrollHeight")
        if new_height == last_height:
            break
        last_height = new_height
    driver.find_element(By.TAG_NAME, 'body').send_keys(Keys.HOME)

driver.get('https://coinmarketcap.com/')

all_tr = driver.find_elements(By.XPATH,'//tbody/tr')

for index, i in enumerate(all_tr):
    make_elements_visible()
    coin_name = driver.find_elements(By.XPATH, '//td[3]/div/a/div/div/p')[index].text
    coin_url = driver.find_elements(By.XPATH, '//td[3]/div/a')[index].get_attribute('href')
    print(coin_name, coin_url)

Output:

Bitcoin https://coinmarketcap.com/currencies/bitcoin/
Ethereum https://coinmarketcap.com/currencies/ethereum/
Tether https://coinmarketcap.com/currencies/tether/
BNB https://coinmarketcap.com/currencies/bnb/
USD Coin https://coinmarketcap.com/currencies/usd-coin/
XRP https://coinmarketcap.com/currencies/xrp/
Solana https://coinmarketcap.com/currencies/solana/
Terra https://coinmarketcap.com/currencies/terra-luna/
Cardano https://coinmarketcap.com/currencies/cardano/
TerraUSD https://coinmarketcap.com/currencies/terrausd/
Binance USD https://coinmarketcap.com/currencies/binance-usd/
Dogecoin https://coinmarketcap.com/currencies/dogecoin/
Avalanche https://coinmarketcap.com/currencies/avalanche/
Polkadot https://coinmarketcap.com/currencies/polkadot-new/
Shiba Inu https://coinmarketcap.com/currencies/shiba-inu/
Wrapped Bitcoin https://coinmarketcap.com/currencies/wrapped-bitcoin/
Dai https://coinmarketcap.com/currencies/multi-collateral-dai/
Polygon https://coinmarketcap.com/currencies/polygon/
NEAR Protocol https://coinmarketcap.com/currencies/near-protocol/
Litecoin https://coinmarketcap.com/currencies/litecoin/
Cronos https://coinmarketcap.com/currencies/cronos/
TRON https://coinmarketcap.com/currencies/tron/
UNUS SED LEO https://coinmarketcap.com/currencies/unus-sed-leo/
FTX Token https://coinmarketcap.com/currencies/ftx-token/
Bitcoin Cash https://coinmarketcap.com/currencies/bitcoin-cash/
Chainlink https://coinmarketcap.com/currencies/chainlink/
Cosmos https://coinmarketcap.com/currencies/cosmos/
Uniswap https://coinmarketcap.com/currencies/uniswap/
Stellar https://coinmarketcap.com/currencies/stellar/
ApeCoin https://coinmarketcap.com/currencies/apecoin-ape/
Algorand https://coinmarketcap.com/currencies/algorand/
Ethereum Classic https://coinmarketcap.com/currencies/ethereum-classic/
Monero https://coinmarketcap.com/currencies/monero/
VeChain https://coinmarketcap.com/currencies/vechain/
Filecoin https://coinmarketcap.com/currencies/filecoin/
Internet Computer https://coinmarketcap.com/currencies/internet-computer/
Hedera https://coinmarketcap.com/currencies/hedera/
Decentraland https://coinmarketcap.com/currencies/decentraland/
Elrond https://coinmarketcap.com/currencies/elrond-egld/
The Sandbox https://coinmarketcap.com/currencies/the-sandbox/
Theta Network https://coinmarketcap.com/currencies/theta-network/
PancakeSwap https://coinmarketcap.com/currencies/pancakeswap/
Tezos https://coinmarketcap.com/currencies/tezos/
THORChain https://coinmarketcap.com/currencies/thorchain/
EOS https://coinmarketcap.com/currencies/eos/
Aave https://coinmarketcap.com/currencies/aave/
Klaytn https://coinmarketcap.com/currencies/klaytn/
Fantom https://coinmarketcap.com/currencies/fantom/
STEPN https://coinmarketcap.com/currencies/green-metaverse-token/
Axie Infinity https://coinmarketcap.com/currencies/axie-infinity/
KuCoin Token https://coinmarketcap.com/currencies/kucoin-token/
Zcash https://coinmarketcap.com/currencies/zcash/
Helium https://coinmarketcap.com/currencies/helium/
Flow https://coinmarketcap.com/currencies/flow/
BitTorrent-New https://coinmarketcap.com/currencies/bittorrent-new/
Huobi Token https://coinmarketcap.com/currencies/huobi-token/
IOTA https://coinmarketcap.com/currencies/iota/
The Graph https://coinmarketcap.com/currencies/the-graph/
Maker https://coinmarketcap.com/currencies/maker/
Bitcoin SV https://coinmarketcap.com/currencies/bitcoin-sv/
eCash https://coinmarketcap.com/currencies/ecash/
TrueUSD https://coinmarketcap.com/currencies/trueusd/
Convex Finance https://coinmarketcap.com/currencies/convex-finance/
Waves https://coinmarketcap.com/currencies/waves/
Nexo https://coinmarketcap.com/currencies/nexo/
Stacks https://coinmarketcap.com/currencies/stacks/
Neo https://coinmarketcap.com/currencies/neo/
Quant https://coinmarketcap.com/currencies/quant/
OKB https://coinmarketcap.com/currencies/okb/
Kusama https://coinmarketcap.com/currencies/kusama/
Celo https://coinmarketcap.com/currencies/celo/
Chiliz https://coinmarketcap.com/currencies/chiliz/
Loopring https://coinmarketcap.com/currencies/loopring/
Harmony https://coinmarketcap.com/currencies/harmony/
Curve DAO Token https://coinmarketcap.com/currencies/curve-dao-token/
Pax Dollar https://coinmarketcap.com/currencies/paxos-standard/
Dash https://coinmarketcap.com/currencies/dash/
Gala https://coinmarketcap.com/currencies/gala/
Neutrino USD https://coinmarketcap.com/currencies/neutrino-usd/
Enjin Coin https://coinmarketcap.com/currencies/enjin-coin/
Zilliqa https://coinmarketcap.com/currencies/zilliqa/
Basic Attention Token https://coinmarketcap.com/currencies/basic-attention-token/
Mina https://coinmarketcap.com/currencies/mina/
NEM https://coinmarketcap.com/currencies/nem/
Arweave https://coinmarketcap.com/currencies/arweave/
Decred https://coinmarketcap.com/currencies/decred/
Amp https://coinmarketcap.com/currencies/amp/
XDC Network https://coinmarketcap.com/currencies/xinfin/
Kava https://coinmarketcap.com/currencies/kava/
Kyber Network Crystal v2 https://coinmarketcap.com/currencies/kyber-network-crystal-v2/
Compound https://coinmarketcap.com/currencies/compound/
Holo https://coinmarketcap.com/currencies/holo/
Theta Fuel https://coinmarketcap.com/currencies/theta-fuel/
Kadena https://coinmarketcap.com/currencies/kadena/
yearn.finance https://coinmarketcap.com/currencies/yearn-finance/
Moonbeam https://coinmarketcap.com/currencies/moonbeam/
PAX Gold https://coinmarketcap.com/currencies/pax-gold/
Gnosis https://coinmarketcap.com/currencies/gnosis-gno/
Secret https://coinmarketcap.com/currencies/secret/
Audius https://coinmarketcap.com/currencies/audius/

Process finished with exit code 0
Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.