Skip to content

Crawling overview

Resources can be crawled using the crawl command. Due to the nature of crawling data on the web, not all data will be present or correct. For that reason it's important to also check out the issues and limitations.

Cards

Depending on the crawler, the card resource will contain complete or partial information. If the card resource contains complete information, the isComplete property will be set to true.

When a card is crawled, it gets automatically analyzed (also see the analyze cards command). If the analyzer generates one or more errors, the isComplete property will be set to false. It's recommended to manually fix these errors and to run the analyze cards command until no more errors are generated. Afterwards (if applicable), the isComplete property can be manually set back to true.

Bulbapedia

Command usage (options)

--bulbapedia-card-list-file-path

In general, Bulbapedia catalogues cards as one or more lists per expansion. However, for some expansions these lists are formatted in a non-standard way, meaning that they can't be properly crawled. In order to crawl these cards anyway, you can use a CSV file containing a custom card list with the following columns:

Column Is value required Description
expansion_name Yes
card_number No Can only be empty if the card has no number.
card_number_sorting_order Yes
rarity No Can only be empty if the card has no rarity.
card_url Yes The Bulbapedia card URL.

CSV headers are required to be present in the file.

CSV file example
expansion_name,card_number,card_number_sorting_order,rarity,card_url
Base Set,1/102,1,Rare Holo,https://bulbapedia.bulbagarden.net/wiki/Alakazam_(Base_Set_1)
Base Set,2/102,2,Rare Holo,https://bulbapedia.bulbagarden.net/wiki/Blastoise_(Base_Set_2)