Skip to content

Creating crawlers

For each comic Comics is aggregating, we need to create a crawler. At the time of writing, more than 200 crawlers are available in the src/comics/comics/ directory. They serve as a great source for learning how to write new crawlers for Comics.

A crawler example

The crawlers are split in two separate pieces. The Metadata part contains meta data about the comic used for display at the web site. The Crawler part contains properties needed for crawling and the crawler implementation itself.

from comics.aggregator.crawler import CrawlerBase, CrawlerImage
from comics.core.metadata import MetadataBase


class Metadata(MetadataBase):
    name = "xkcd"
    language = "en"
    url = "https://www.xkcd.com/"
    start_date = "2005-05-29"
    rights = "Randall Munroe, CC BY-NC 2.5"


class Crawler(CrawlerBase):
    history_length_days = 10
    schedule = "Mo,We,Fr"
    time_zone = "US/Eastern"

    def crawl(self, pub_date):
        feed = self.parse_feed("https://www.xkcd.com/rss.xml")
        for entry in feed.for_date(pub_date):
            url = entry.summary.src('img[src*="/comics/"]')
            title = entry.title
            text = entry.summary.alt('img[src*="/comics/"]')
            return CrawlerImage(url, title, text)

The Metadata class

comics.core.metadata.MetadataBase dataclass 

MetadataBase()

Base class for the metadata part of a crawler module.

Each crawler module must define a subclass of this class named Metadata, overriding the class attributes to describe the comic. The metadata is used for display at the web site.

Attributes:

  • name (str) –

    Required. A string with the name of the comic.

  • language (str) –

    Required. A two-letter string with the language code for the language

  • url (str) –

    Required. A string with the URL of the comic’s web page.

  • active (bool) –

    Optional. Whether or not this comic is still being crawled.

  • start_date (str | None) –

    Optional. The first date the comic was published at, as an

  • end_date (str | None) –

    Optional. The last date the comic was published at, as an ISO 8601

  • rights (str) –

    Optional. Name of the author and the comic’s license if available.

  • slug (str) –

    The comic’s slug, used in URLs and to identify the comic.

name class-attribute instance-attribute 

name: str = field(init=False)

Required. A string with the name of the comic.

language class-attribute instance-attribute 

language: str = field(init=False)

Required. A two-letter string with the language code for the language used in the comic. Typically "en" or "no".

The language code must also be present in comics.core.models.Comic.LANGUAGES.

url class-attribute instance-attribute 

url: str = field(init=False)

Required. A string with the URL of the comic’s web page.

active class-attribute instance-attribute 

active: bool = field(init=False, default=True)

Optional. Whether or not this comic is still being crawled.

Defaults to True.

start_date class-attribute instance-attribute 

start_date: str | None = field(init=False, default=None)

Optional. The first date the comic was published at, as an ISO 8601 date string, e.g. "2005-05-29".

end_date class-attribute instance-attribute 

end_date: str | None = field(init=False, default=None)

Optional. The last date the comic was published at, as an ISO 8601 date string, if the comic is discontinued.

rights class-attribute instance-attribute 

rights: str = field(init=False, default='')

Optional. Name of the author and the comic’s license if available.

slug class-attribute instance-attribute 

slug: str = field(init=False)

The comic’s slug, used in URLs and to identify the comic.

Set automatically to the crawler module’s file name, so it cannot be overridden.

The Crawler class

comics.aggregator.crawler.CrawlerBase dataclass

Base class for the crawler part of a crawler module.

Each crawler module must define a subclass of this class named Crawler, overriding the class attributes as needed and implementing the crawl() method.

Methods:

  • crawl –

    Crawl the comic’s site for the release published on pub_date.

  • parse_feed –

    Fetch and parse the RSS or Atom feed at feed_url.

  • parse_page –

    Fetch and parse the web page at page_url.

  • string_to_date –

    Parse string as a date, using a strptime() format string.

  • date_to_epoch –

    The UNIX time of midnight at date in the comic’s time zone.

Attributes:

  • history_start_date (str | None) –

    Optional. Date of oldest release available for crawling, as an

  • history_length_days (int | None) –

    Optional. Number of days a release is available for crawling, e.g.

  • schedule (str | None) –

    Optional. On what weekdays the comic is published.

  • time_zone (str) –

    Optional. In approximately what time zone the comic is published.

  • multiple_releases_per_day (bool) –

    Optional. Whether to allow multiple releases per day.

  • has_rerun_releases (bool) –

    Optional. Whether the comic reruns old images as new releases.

  • headers (RequestHeaders) –

    Optional. Any HTTP headers to send with any URL request, both when

history_start_date class-attribute instance-attribute 

history_start_date: str | None = field(init=False, default=None)

Optional. Date of oldest release available for crawling, as an ISO 8601 date string, e.g. "2008-03-08".

Provide this or history_length_days. If both are present, this one will have precedence.

history_length_days class-attribute instance-attribute 

history_length_days: int | None = field(init=False, default=None)

Optional. Number of days a release is available for crawling, e.g. 32.

Provide this or history_start_date.

schedule class-attribute instance-attribute 

schedule: str | None = field(init=False, default=None)

Optional. On what weekdays the comic is published.

Example: "Mo,We,Fr" or "Mo,Tu,We,Th,Fr,Sa,Su".

time_zone class-attribute instance-attribute 

time_zone: str = field(init=False, default='UTC')

Optional. In approximately what time zone the comic is published.

Example: "Europe/Oslo" or "US/Eastern". See the IANA timezone database for a list of possible values. Defaults to "UTC".

multiple_releases_per_day class-attribute instance-attribute 

multiple_releases_per_day: bool = field(init=False, default=False)

Optional. Whether to allow multiple releases per day.

Defaults to False.

has_rerun_releases class-attribute instance-attribute 

has_rerun_releases: bool = field(init=False, default=False)

Optional. Whether the comic reruns old images as new releases.

Defaults to False.

headers class-attribute instance-attribute 

headers: RequestHeaders = field(default_factory=dict)

Optional. Any HTTP headers to send with any URL request, both when crawling and when downloading images.

Useful if you’re pulling comics from a site that checks either the Referer or User-Agent. If you can view the comic using your browser but not when using your crawler for identical URLs, try setting Referer to the comic’s site or User-Agent to a browser’s user agent string.

Example: {"Referer": "http://www.example.com/"}.

crawl 

crawl(pub_date: date) -> CrawlerResult

Crawl the comic’s site for the release published on pub_date.

Must be overridden by all crawlers.

On success, the returned CrawlerImage contains at least the URL of the image, and optionally a title and/or a text accompanying the image.

If the release consists of multiple images, return a list of CrawlerImage objects, ordered in the same way as the comic is meant to be read, with the first frame as the first element in the list.

Parameters:

  • pub_date
    (date) –

    The date to crawl.

Returns:

  • CrawlerResult –

    The crawled image, a list of images, or None if no release was published on pub_date.

parse_feed 

parse_feed(feed_url: str) -> FeedParser

Fetch and parse the RSS or Atom feed at feed_url.

The returned FeedParser is cached and reused when crawling multiple dates.

parse_page 

parse_page(page_url: str) -> LxmlParser

Fetch and parse the web page at page_url.

The returned LxmlParser is cached per URL and reused when crawling multiple dates.

string_to_date 

string_to_date(string: str, fmt: str) -> date

Parse string as a date, using a strptime() format string.

date_to_epoch 

date_to_epoch(date: date) -> int

The UNIX time of midnight at date in the comic’s time zone.

The Crawler.crawl() method

The Crawler.crawl() is where the real work is going on. To start with an example, let’s look at XKCD’s Crawler.crawl() method:

def crawl(self, pub_date):
    feed = self.parse_feed("http://www.xkcd.com/rss.xml")
    for entry in feed.for_date(pub_date):
        url = entry.summary.src('img[src*="/comics/"]')
        title = entry.title
        text = entry.summary.alt('img[src*="/comics/"]')
        return CrawlerImage(url, title, text)

Arguments and return values

The Crawler.crawl() method takes a single argument, pub_date, which is a datetime.date object for the date the crawler is currently crawling. The goal of the method is to return a CrawlerImage object containing at least the URL of the image for pub_date and optionally a title and text accompanying the image.

For some crawlers, this is all you need. If the image URL is predictable and based upon the pub_date in some way, just create the URL with the help of Python’s strftime documentation, and return it wrapped in a CrawlerImage:

def crawl(self, pub_date):
    url = "http://www.example.com/comics/%s.png" % (pub_date.strftime("%Y-%m-%d"),)
    return CrawlerImage(url)

Though, for most crawlers, some interaction with RSS or Atom feeds or web pages are needed. For this a web parser and a feed parser are provided.

Returning multiple images for a single comic release

Some comics got releases with multiple images, and thus returning a single CrawlerImage will not be enough for you. For situations like these, Comics lets you return a list of CrawlerImage objects from Crawler.crawl(). The list should be ordered in the same way as the comic is meant to be read, with the first frame as the first element in the list. If the comic release got a title, add it to the first CrawlerImage object, and let the title field stay empty on the rest of the list elements. The same applies for the text field, unless each image actually got a different title or text string.

The following is an example of a Crawler.crawl() method which returns multiple images. It adds a title to the first list element, and different text to all of the elements.

def crawl(self, pub_date):
    feed = self.parse_feed("http://feeds.feedburner.com/Pidjin")
    for entry in feed.for_date(pub_date):
        result = []
        for i in range(1, 10):
            url = entry.content0.src('img[src$="000%d.jpg"]' % i)
            text = entry.content0.title('img[src$="000%d.jpg"]' % i)
            if url and text:
                result.append(CrawlerImage(url, text=text))
        if result:
            result[0].title = entry.title
        return result

The CrawlerImage class

comics.aggregator.crawler.CrawlerImage dataclass 

CrawlerImage

A single image in a comic release, as returned by crawlers.

A crawler must always supply an URL, while title and text are optional and independent of each other. The following are all valid ways to create a CrawlerImage:

CrawlerImage(url)
CrawlerImage(url, title)
CrawlerImage(url, title, text)
CrawlerImage(url, text=text)

Attributes:

  • url (str | None) –

    The URL of the comic image.

  • title (str | None) –

    An optional title accompanying the image.

  • text (str | None) –

    An optional text accompanying the image.

url instance-attribute 
url: str | None

The URL of the comic image.

title class-attribute instance-attribute 
title: str | None = None

An optional title accompanying the image.

text class-attribute instance-attribute 
text: str | None = None

An optional text accompanying the image.

Parsing web pages with LxmlParser

The web parser, internally known as LxmlParser, uses CSS selectors to extract content from HTML. For a primer on CSS selectors, see Matching HTML elements using CSS selectors.

The web parser is accessed through the Crawler.parse_page method:

def crawl(self, pub_date):
    page_url = "http://ars.userfriendly.org/cartoons/?id=%s" % (
        pub_date.strftime("%Y%m%d"),
    )
    page = self.parse_page(page_url)
    url = page.src('img[alt^="Strip for"]')
    return CrawlerImage(url)

This is a common pattern for crawlers. Another common patterns is to use a feed to find the web page URL for the given date, then parse that web page to find the image URL.

LxmlParser API

comics.aggregator.lxmlparser.LxmlParser 

LxmlParser(
    url: str | None = None,
    string: str | None = None,
    headers: dict[str, str] | None = None,
)

Parser for web pages and HTML fragments, using CSS selectors.

The parser is initialized with either a url to fetch and parse, or a string of HTML to parse. Relative URLs in the document are automatically expanded to absolute URLs, so e.g. a src of /comics/2008-04-13.png is returned as http://www.example.com/comics/2008-04-13.png.

All extraction methods take a CSS selector to match elements. In the event that the selector doesn’t match any elements, default is returned.

If the selector matches multiple elements, one of two things will happen:

  • Singular methods, e.g. src(), raise a MultipleElementsReturned exception.
  • Plural methods, e.g. srcs(), return a list of zero or more values.

Methods:

  • text –

    Return the text contained by the element matching selector.

  • texts –

    Return a list of the text contained by the elements matching selector.

  • src –

    Return the src attribute of the element matching selector.

  • srcs –

    Return the src attribute of the elements matching selector.

  • alt –

    Return the alt attribute of the element matching selector.

  • alts –

    Return the alt attribute of the elements matching selector.

  • title –

    Return the title attribute of the element matching selector.

  • titles –

    Return the title attribute of the elements matching selector.

  • href –

    Return the href attribute of the element matching selector.

  • hrefs –

    Return the href attribute of the elements matching selector.

  • value –

    Return the value attribute of the element matching selector.

  • values –

    Return the value attribute of the elements matching selector.

  • id –

    Return the id attribute of the element matching selector.

  • ids –

    Return the id attribute of the elements matching selector.

  • attr –

    Return the given attr attribute of the element matching selector.

  • attrs –

    Return the given attr attribute of the elements matching selector.

  • content –

    Return the content attribute of the element matching selector.

  • contents –

    Return the content attribute of the elements matching selector.

  • remove –

    Remove the elements matching selector from the parsed document.

  • url –

    Return the URL of the parsed page, after following any redirects.

text 
text(selector: str, *, default: str) -> str

text(selector: str, *, default: str | None = ...) -> str | None

text(selector: str, *, default: str | None = None) -> list[str] | str | None

Return the text contained by the element matching selector.

texts 
texts(selector: str) -> list[str]

Return a list of the text contained by the elements matching selector.

src 
src(selector: str, *, default: str) -> str

src(selector: str, *, default: str | None = None) -> str | None

src(selector: str, *, default: str | None = None) -> str | None

Return the src attribute of the element matching selector.

srcs 
srcs(selector: str) -> list[str]

Return the src attribute of the elements matching selector.

alt 
alt(selector: str, *, default: str) -> str

alt(selector: str, *, default: str | None = None) -> str | None

alt(selector: str, *, default: str | None = None) -> str | None

Return the alt attribute of the element matching selector.

alts 
alts(selector: str) -> list[str]

Return the alt attribute of the elements matching selector.

title 
title(selector: str, *, default: str) -> str

title(selector: str, *, default: str | None = None) -> str | None

title(selector: str, *, default: str | None = None) -> str | None

Return the title attribute of the element matching selector.

titles 
titles(selector: str) -> list[str]

Return the title attribute of the elements matching selector.

href 
href(selector: str, *, default: str) -> str

href(selector: str, *, default: str | None = None) -> str | None

href(selector: str, *, default: str | None = None) -> str | None

Return the href attribute of the element matching selector.

hrefs 
hrefs(selector: str) -> list[str]

Return the href attribute of the elements matching selector.

value 
value(selector: str, *, default: str) -> str

value(selector: str, *, default: str | None = None) -> str | None

value(selector: str, *, default: str | None = None) -> str | None

Return the value attribute of the element matching selector.

values 
values(selector: str) -> list[str]

Return the value attribute of the elements matching selector.

id 
id(selector: str, *, default: str) -> str

id(selector: str, *, default: str | None = None) -> str | None

id(selector: str, *, default: str | None = None) -> str | None

Return the id attribute of the element matching selector.

ids 
ids(selector: str) -> list[str]

Return the id attribute of the elements matching selector.

attr 
attr(attr: str, selector: str, *, default: str) -> str

attr(attr: str, selector: str, *, default: str | None = None) -> str | None

attr(attr: str, selector: str, *, default: str | None = None) -> str | None

Return the given attr attribute of the element matching selector.

attrs 
attrs(attr: str, selector: str) -> list[str]

Return the given attr attribute of the elements matching selector.

content 
content(selector: str, *, default: str) -> str

content(selector: str, *, default: str | None = None) -> str | None

content(selector: str, *, default: str | None = None) -> str | None

Return the content attribute of the element matching selector.

contents 
contents(selector: str) -> list[str]

Return the content attribute of the elements matching selector.

remove 
remove(selector: str) -> None

Remove the elements matching selector from the parsed document.

url 
url() -> str | None

Return the URL of the parsed page, after following any redirects.

Matching HTML elements using CSS selectors

Both web page and feed parsing uses CSS selectors to extract the interesting strings from HTML. CSS selectors are those normally simple strings you use in CSS style sheets to select what elements of your web page the CSS declarations should be applied to.

In the following example h1 a is the selector. It matches all a elements contained in h1 elements. The rule to be applied to the matching elements is color: red;.

h1 a { color: red; }

Similarly class="foo" and id="bar" in HTML may be used in CSS selectors. The following CSS example would color all h1 headers with the class foo red, and all elements with the ID bar which is contained in h1 elements would be colored blue.

h1.foo { color: red; }
h1 #bar { color: blue; }

In CSS3, the power of CSS selectors have been greatly increased by the addition of matching by the content of elements’ attributes. To match all img elements with a src attribute starting with http://www.example.com/ simply write:

img[src^="http://www.example.com/"]

Or, to match all img elements whose src attribute ends in .jpg:

img[src$=".jpg"]

Or, img elements whose src attribute contains /comics/:

img[src*="/comics/"]

Or, img elements whose alt attribute is Today's comic:

img[alt="Today's comic"]

For further details on CSS selectors in general, please refer to http://css.maxdesign.com.au/selectutorial/.

Parsing feeds with FeedParser

The feed parser is initialized with a feed URL passed to Crawler.parse_feed, just like the web parser is initialized with a web page URL:

def crawl(pub_date):
    ...
    feed = self.parse_feed("http://www.xkcd.com/rss.xml")
    ...

FeedParser API

The feed object provides two methods which both returns feed entries: FeedParser.for_date and FeedParser.all. Typically, a crawler uses FeedParser.for_date and loops over all entries it returns to find the image URL:

for entry in feed.for_date(pub_date):
    # parsing comes here
    return CrawlerImage(url)

comics.aggregator.feedparser.FeedParser 

FeedParser(url: str)

Parser for RSS and Atom feeds.

The parser is initialized with the feed URL to fetch and parse, typically through Crawler.parse_feed().

Methods:

  • for_date –

    Return all feed entries published or updated at date.

  • all –

    Return all feed entries.

for_date 
for_date(date: date) -> list[Entry]

Return all feed entries published or updated at date.

all 
all() -> list[Entry]

Return all feed entries.

Feed Entry API

comics.aggregator.feedparser.Entry 

Entry(entry: FeedParserDict, encoding: str | None = None)

A feed entry, as returned by the feed parser.

This is really a combination of the popular feedparser library and LxmlParser. It can do anything feedparser can do, and in addition you can use the LxmlParser methods on feed fields which contain HTML, like summary and content0.

Methods:

Attributes:

summary instance-attribute 
summary: LxmlParser

The entry’s summary, with LxmlParser methods available for HTML parsing.

This is the most frequently used entry field. Example usage:

url = entry.summary.src("img")
title = entry.summary.alt("img")

Only present if the feed entry has a summary.

content0 instance-attribute 
content0: LxmlParser

The same as feedparser’s content[0].value field, but with LxmlParser methods available for HTML parsing.

For some crawlers, this is where the interesting stuff is found. Only present if the feed entry has content.

tags property 
tags: list[str]

List of tags associated with the entry.

html 
html(value: str | bytes) -> LxmlParser

Wrap value in a LxmlParser.

If you need to parse HTML in any other fields than summary and content0, you can apply this method on the field, like it is applied on a feed entry’s title field here:

title = entry.html(entry.title).text("h1")

Testing your new crawler

When the first version of you crawler is complete, it’s time to test it.

The file name is important, as it is used as the comic’s slug. This means that it must be unique within the Comics installation, and that it is used in the URLs Comics will serve the comic at. For this example, we call the crawler file foo.py. The file must be placed in the src/comics/comics/ directory, and will be available in Python as comics.comics.foo.

Loading Metadata for your new comic

For Comics to know about your new crawler, you need to load the comic meta data into Comics’ database. To do so, we run the add_comics command:

uv run comics add_comics -c foo

If you do any changes to the Metadata class of any crawler, you must rerun add_comics to update the database representation of the comic.

Running the crawler

When add_comics has created a comics.core.models.Comic instance for the new crawler, you may use your new crawler to fetch the comic’s release for the current date by running:

uv run comics get_releases -c foo

If you want to get comics releases for more than the current day, you may specify a date range to crawl, like:

uv run comics get_releases -c foo -f 2009-01-01 -t 2009-03-31

The date range will automatically be adjusted to the crawlers history capability. You may also get comics for a date range without a specific end. In which case, the current date will be used instead:

uv run comics get_releases -c foo -f 2009-01-01

If your new crawler is not working properly, you may add -v2 to the command to turn on full debug output:

uv run comics get_releases -c foo -v2

For a full overview of get_releases options, run:

uv run comics get_releases --help

Submitting your new crawler for inclusion in Comics

When your crawler is working properly, you may submit it for inclusion in Comics. You should fork Comics at GitHub, commit your new crawler to your own fork, and send me a pull request through GitHub.

All contributions must be granted under the same license as Comics itself.