Crawling TCGplayer card variant prices
This document provides information for the TCGplayer crawler of the crawl card-variant-prices command.
TCGplayer has somewhat limited support for card variants. They have a fixed set of card variant types, which they call printings. Each product (card) on TCGplayer is assigned one or more of these printings, from which sellers can choose. The available printings are:
- Normal
- Holofoil
- Reverse Holofoil
- Unlimited
- Unlimited Holofoil
- 1st Edition
- 1st Edition Holofoil
The Unlimited and Unlimited Holofoil printings are identical to the Normal and Holofoil printings. They're used exclusively for products that also have 1st Edition or 1st Edition Holofoil printings. This is to avoid confusion for buyers who might incorrectly assume Normal or Holofoil printings to be 1st Edition or 1st Edition Holofoil printings.
In the context of crawling card variant prices, this distinction for the Unlimited and Unlimited Holofoil printings has no real use. Therefore, we don't use them and automatically translate them to Normal and Holofoil internally.
TCGplayer has no option for card variants like Cracked Ice Holo or Cosmos Holo. This means that they're forced to list these card variants as separate products using a best matching printing (usually Normal or Holofoil). This makes it extremely difficult to automatically figure out which card variant they're referring to. Our solution is to use an external database that contains this mapping information.
A lot of card names, expansion names and card numbers are labeled in an inconsistent manner. For that reason, mapping information for these values is included in the external database as well.
Command usage (options)
--tcgplayer-product-mappings-dir-path
Because TCGplayer is quite inconsistent when it comes to card names, expansion names and card variants (as explained previously), an external database is needed that maps the values as they appear on TCGplayer to their corrected/sanitized values. This database consists out of a JSON file per product with the following properties:
| Property | Type | Is required | Description |
|---|---|---|---|
| productId | int32 | Yes | The TCGplayer product ID. |
| productName | string | Yes | The TCGplayer product name. It usually refers to the card name. |
| productGroupName | string | Yes | The TCGplayer group name. It usually refers to the expansion name. |
| productNumber | string|null | Yes | The TCGplayer product number. It usually refers to the card number. |
| productUrl | string | Yes | The TCGplayer product URL. |
| cardId | string|null | No | If this value is set, all other properties will be ignored when identifying the card. This can be useful when mapping card (variant) prices to an external database such as TCG Collector. |
| cardName | string|null | No | Optionally can be used to help identify the card. |
| expansionName | string | Yes | |
| cardNumber | string|null | Yes | |
| normalCardVariantId | string|null | No | See 1*. |
| holofoilCardVariantId | string|null | No | See 1*. |
| reverseHolofoilCardVariantId | string|null | No | See 1*. |
| firstEditionCardVariantId | string|null | No | See 1*. |
| firstEditionHolofoilCardVariantId | string|null | No | See 1*. |
| normalCardVariantType | string | Yes | See 2*. |
| holofoilCardVariantType | string | Yes | See 2*. |
| reverseHolofoilCardVariantType | string | Yes | See 2*. |
| firstEditionCardVariantType | string | Yes | See 2*. |
| firstEditionHolofoilCardVariantType | string | Yes | See 2*. |
This option is not required because no product mappings will exist yet on the first run. After the first run, the product mappings can be created by mapping the unhandled products output (see the --tcgplayer-unhandled-products-output-dir-path option).
Each JSON file is required to have a tcgplayer-product-mapping suffix, e.g., 158963.tcgplayer-product-mapping.json.
1*: If this value is set, all other properties will be ignored when identifying the card variant. This can be useful when mapping card (variant) prices to an external database such as TCG Collector.
2*: The *cardVariantType properties map the different TCGplayer printings to card variant types. All properties are required to have a value because TCGplayer doesn't always list the correct printings for a product.
--tcgplayer-skipped-products-dir-path
Sometimes, TCGplayer will list products that you don't want to track. In order to prevent these products from showing up in the unhandled products output (see the --tcgplayer-unhandled-products-output-dir-path option), a JSON file per product with the following properties can be used:
| Property | Type | Is required | Description |
|---|---|---|---|
| id | int32 | Yes | The TCGplayer product ID. |
| name | string | Yes | The TCGplayer product name. It usually refers to the card name. |
| groupName | string | Yes | The TCGplayer group name. It usually refers to the expansion name. |
| number | string|null | Yes | The TCGplayer product number. It usually refers to the card number. |
| url | string | Yes | The TCGplayer product URL. |
| reasonForSkipping | string|null | No | The reason for skipping the TCGplayer product. |
Each JSON file is required to have a tcgplayer-skipped-product suffix, e.g., 158963.tcgplayer-skipped-product.json.
--tcgplayer-unhandled-products-output-dir-path (required)
Unhandled products are TCGplayer products for which no product mapping could be found during the crawling process. Each unhandled product is exported as a JSON file with the following properties:
| Property | Type | Is required | Description |
|---|---|---|---|
| id | int32 | Yes | The TCGplayer product ID. |
| name | string | Yes | The TCGplayer product name. It usually refers to the card name. |
| groupName | string | Yes | The TCGplayer group name. It usually refers to the expansion name. |
| number | string|null | Yes | The TCGplayer product number. It usually refers to the card number. |
| url | string | Yes | The TCGplayer product URL. |
This output should be mapped manually into product mappings or skipped products. If the TCG Collector database is used, the unhandled products can be semi-automatically mapped using the cms map tcgplayer-unhandled-products command.
To get an overview of all unhandled TCGplayer groups, the show tcgplayer-unhandled-groups command can be used.
Each JSON file is required to have a tcgplayer-unhandled-product suffix, e.g., 158963.tcgplayer-unhandled-product.json.
--tcgplayer-api-config-file-path (required)
This JSON file contains the configuration for the TCGplayer API.
| Property | Type | Is required | Description |
|---|---|---|---|
| publicKey | string | Yes | |
| privateKey | string | Yes |
How to map TCGplayer printings to card variants
For non-holographic products that are not a card variant listed as a separate product:
| Property | TCGplayer printing | Card variant type | Notes |
|---|---|---|---|
| normalCardVariantType | Normal | Standard Set | |
| holofoilCardVariantType | Holofoil | Parallel Set: Reverse Foil | See 1*. |
| reverseHolofoilCardVariantType | Reverse Holofoil | Parallel Set: Reverse Foil | See 1*. |
| firstEditionCardVariantType | 1st Edition | Standard Set | |
| firstEditionHolofoilCardVariantType | 1st Edition Holofoil | Parallel Set: Reverse Foil | See 1*. |
For holographic products that are not a card variant listed as a separate product:
| Property | TCGplayer printing | Card variant type | Notes |
|---|---|---|---|
| normalCardVariantType | Normal | Standard Set Foil | |
| holofoilCardVariantType | Holofoil | Standard Set Foil | |
| reverseHolofoilCardVariantType | Reverse Holofoil | Parallel Set: Reverse Foil | See 1*. |
| firstEditionCardVariantType | 1st Edition | Standard Set Foil | |
| firstEditionHolofoilCardVariantType | 1st Edition Holofoil | Standard Set Foil |
For products that are a card variant listed as a separate product, the mappings will have to be adjusted according to the card variant type. Take Cosmos Holo for example:
| Property | TCGplayer printing | Card variant type |
|---|---|---|
| normalCardVariantType | Normal | Standard Set |
| holofoilCardVariantType | Holofoil | Cosmos Holo |
| reverseHolofoilCardVariantType | Reverse Holofoil | Cosmos Holo |
| firstEditionCardVariantType | 1st Edition | Standard Set |
| firstEditionHolofoilCardVariantType | 1st Edition Holofoil | Cosmos Holo |
1*: A card that doesn't have a Parallel Set: Reverse Foil variant might have an additional variant that is holographic, such as Cosmos Holo. If so, the TCGplayer printing should map to this card variant instead of Parallel Set: Reverse Foil. If the card has multiple holographic variants that are not Parallel Set: Reverse Foil, it will need to be investigated manually.
Automatic mapping
When using the cms map tcgplayer-unhandled-products or cms map tcgplayer-product-mappings command, there is a runtime option to attempt to automatically map the TCGplayer printings. However, this option is not completely foolproof and should not be used in the following cases:
-
The TCGplayer product is a separately listed card variant.
Automatic mapping only works for "main" TCGplayer products and "main" products can unfortunately not be automatically distinguished from "card variant" products.
-
The TCG Collector card variants are incomplete.
The mapping of the TCGplayer printings differs based on which TCG Collector card variants are available. If automatic mapping proceeds with an incomplete list of card variants, duplicate price errors are likely to occur during the crawling process (also see the Changes to product mappings over time section).
-
The TCG Collector card rarity is undetermined or incorrect.
The TCG Collector card rarity is used to determine whether the card is holographic or not. This directly affects how the TCGplayer printings are mapped.
Changes to product mappings over time
Over time, manual maintenance to the product mappings might be required because of the following reasons:
-
A new product has been added on TCGplayer.
See the
--tcgplayer-unhandled-products-output-dir-pathoption. -
A new printing has been added to an existing TCGplayer product.
A new printing will cause the existing product mapping to incorrectly map to the same card variant. The crawler will log an error about a duplicate card variant that will need to be corrected manually. Although rare, it's also possible a seller on TCGplayer specified an incorrect printing. In that case, the error can be ignored.
If the TCG Collector database is used, existing product mappings (including skipped products) can be semi-automatically remapped using the
cms map tcgplayer-product-mappingscommand. -
A product has been deleted on TCGplayer.
This deleted product should remain in the product mappings, because there is a chance of the product becoming available again. The crawler doesn't log any warnings or errors about deleted products.
-
A new card (variant) has been added to an externally mapped database.
If the card (variant) doesn't exist on TCGplayer, no action is required. If TCGplayer adds this card (variant) at one point, see item 1 and 2.
-
One or more of the values of the mappings have gone out of sync with an externally mapped database.
Unfortunately, the crawler has no way of detecting values that have gone out of sync with an externally mapped database. However, it's advised that the process of uploading the card (variant) prices to externally mapped databases should log warnings about it. This primarily concerns changes and deletions, since new values are already automatically detected (see item 4). With the help of these logs, it should be possible to correct the mappings manually.