Command-line tool for spidering sites and extracting XML/HTML content

Xidel is a robust tool for spidering, extracting and transforming XML/HTML content from the command line.

It's like wget or curl with a CSS and XPath/XQuery engine (among other features), attached.

xidel doesn't seem to be in the package management repositories I normally use, but you can download it here.

The following example will (1) download a web page, (2) extract a list of links (specified via CSS selector) from it, (3) download the page corresponding to each of those links and finally (4) extract specific pieces of content (specified by CSS selectors) from each page:

xidel [URL-OF-INDEX-PAGE] \
  --follow "css('[CSS-SELECTOR-FOR-LINKS]')" \
  --css "[CSS-SELECTOR-FOR-SOME-TEXT]" \
  --extract "inner-html(css('[CSS-SELECTOR-FOR-SOME-HTML]'))"

As a concrete example, the command:

$ xidel http://reddit.com -f  "css('a')" --css title

will download every page linked from the reddit.com homepage and print the content of its title tag.

There are several more examples on the Xidel site.

Cross-browser CSS transitions

.foo {
  transition:         opacity .25s ease-in-out;
  -webkit-transition: opacity .25s ease-in-out;
  -moz-transition:    opacity .25s ease-in-out;
  -o-transition:      opacity .25s ease-in-out;
}

Where the right-hand side has the general form:

PROPERTY DURATION [TIMING-FUNCTION] [TRANSITION-DELAY]

Transitions can chained:

 .foo { transition: opacity .25s ease-in-out, height 1s linear }

Transition timing functions

• linear - constant rate of change between states.

  cubic-bezier(0.0, 0.0, 1.0, 1.0)

• ease - (the default) slow acceleration, then faster, before rapidly slowing again at the end.

  cubic-bezier(0.25, 0.1, 0.25, 1.0)

• ease-in-out - like ease but accelerating/decelerating more rapidly (with a shorter transition between acceleration and deceleration).

  cubic-bezier(0.42, 0, 0.58, 1.0)

• ease-in - equivalent to the first half of ease-in-out; rapid accelerating then transitioning to a constant rate of change at the end.

  cubic-bezier(0.42, 0, 1, 1.0)

• ease-out - equivalent to the second half of ease-in-out; a constant rate of change transitioning rapid deceleration at the end.

  cubic-bezier(0.42, 0, 0.58, 1.0)

• cubic-bezier(x1,y1,x2,y2) - follows cubic Bézier curve using the control points (0,0), (x1,y1), (x2,y2) and (1,1).

• steps( n, [start|end] ) - stepwise function with n steps

How to read cubic-bezier functions

Imagine the transition as a stepwise function starting at (0,0) and ending at (1,1) and with two steps in between. The cubic-bezier(x1,y1,x2,y2) function specifies the points in the middle. In other words, x1 and x2 are the times at which the second and third steps happen, respectively (expressed as a fraction of the total transition duration) and y1 and y2 are how far along the transition is at x1 and x2, respectively (expressed as a fraction of the total transition "distance"). Very loosely, the cubic-bezier function "smooths" those steps.

These are nicely demonstrated here.

Presentational CSS Classes

These are the opposite of semantic markup, but I find them useful:

.align-both   { text-align: justify; }
.align-center { text-align: center; }
.align-left   { text-align: left; }
.align-right  { text-align: right; }
.float-left   { float: left; }
.float-right  { float: right; }
.nobr         { white-space: nowrap; }
.small        { font-size: 80%; } /* should really sync with the <small> tag rules */

Fixing CSS Resets

CSS reset frameworks often strip out all formatting. These CSS rules contain some re-resets:

/* I understand the theory behind replacing <i> and <b> with <em> and <strong>, but c'mon, really? */
i     { font-style: italic; }
b     { font-weight: bold; }
small { font-size: 80%; }

/* Make superscripts and subscripts actually do something: */
sup, sub { height: 0; line-height: 1; vertical-align: baseline; _vertical-align: bottom; position: relative; }
sup      {    bottom: 1ex; }
sub      {    top: .5ex; }

/* Mono-spaced types. */
pre, code, kbd, samp, tt { font-family: 'droid sans mono slashed', 'droid sans mono dotted', 'droid sans mono', monospace, monospace; }

Vertically centering block elements with CSS.

Via phrogz.net.

<style type="text/css">
    #wrapper { position:relative; } /* position:absolute is also ok */
    #wrapped { position:absolute; top:50%; height:10em; margin-top:-5em; }  /* margin-top = -1 * height/2 */
</style>
...
<div id="wrapper">
    <div id="wrapped">
        <p>Block content.</p>
        <p>But still vertically centered.</p>
    </div>
</div>