indie-org ¶

2.1.1 Microformats ¶

Microformats are a fundamental building block of the Indieweb. “Designed for humans first and machines second, microformats are a set of simple, open data formats built upon existing and widely adopted standards”. They are a set of tags which one may add to one’s HTML to indicate semantic rather than formatting information. For instance, consider mentioning a person in your prose. In plain text, one could say:

Barnaby Walters

and in HTML, to indicate a location on the internet, one could say:

<a href="http://waterpigs.co.uk">Barnaby Walters</a>

Microformats2 adds additional information to that link, the information that this names a person, by adding a particular class to the HTML element:

<a class="h-card" href="http://waterpigs.co.uk">Barnaby Walters</a>

Microformats allow you to build structure, for instance giving this reference to a person both a location on the web and a photo:

<div class="h-card">
    <p><img class="u-photo" href="/me.png" alt="" /></p>
    <p class="p-name"><a href="u-url" href="http://waterpigs.co.uk">Barnaby Walters</a></p>
</div>

You can find a complete vocabulary here, and an on-line Microformats parser here.

Of particular note is that, as static site authors, the onus is on us to mark-up our HTML with microformts. We’ll use the page template in the Org Export back-end to do so (see below).

The Microformats Wiki is here and mf2py is a commonly used Python library for parsing Microformats.

2.1.2 IndieAuth ¶

In a world in which everyone owns their own identity, how shall we authenticate one another? OAuth 2.0 has become increasingly common, but suffers from a few shortcomings that make it unsuitable for use on the Indieweb.

In the first place, the protocol requires an app to register with the authorization server to be used (see the RFC, section 2). This makes sense when, say, building a Facebook app, or a Twitter app, but breaks-down in a world in which everyone is their own authorizing authority.

The second issue is simply that OAuth 2.0 is not an authentication protocol– it is a protocol designed to authorize applications to access resources on a principal’s behalf. Without augmenting the protocol in some way, the app will have no information that can identify the user on their app– just a grant of permissions to carry out certain operations on the resource provider.

This led Aaron Parecki and others to develop a related protocol– IndieAuth. Here, your identity is your domain name. When you wish to authenticate with a site, you no longer pick one of a few silos (“sign-in with Google”, “sign-in with Facebook”, and so forth), but instead provide your domain name. The site will fetch your page (or, at least, its head) to discover your IndieAuth provider endpoints.

From there, the flow largely proceeds in the same manner as OAuth with the additional bit that the authorization endpoint will provide not only a token, but a canonical URL identifying the user. The full authorization flow is described here.

This is an important step forward, but making everyone on the Indieweb responsible for implementating an authentication server would not be very practical. Fortunately, there are multiple standalone implementations available, beginning with IndieAuth.com. This is an IndieAuth authentication implementation that will allow one to authenticate using either one’s existing social media accounts or PGP keypair, even (or, perhaps, especially) when using a static site.

In order to integrate with IndieAuth.com, the site author needs to add a link to each account to the home page with the attribute rel="me", and ensure that those accounts list your home page in their profiles:

<ul>
  <li>
    <a href="https://twitter.com/jdoe" rel="me">Twitter</a>
  </li>
  <li>
    <a href="https://github.com/jdoe" rel="me">Github</a>
  </li>
  ...
</ul>

This manual will cover the details for a sample site See below.

2.1.3 Webmentions ¶

Webmention is an open standard for distributed interactions among independent web sites.

Suppose Alice posts to her site. The content is presumably HTML, but could be something else (plain/text, say). Bob now wants to mention that content on his site. He makes a post (the source) that contains the URL of Alice’s post (the target). Again, the source is presumably an HTML document, but may be anything that can express an URL.

Bob, or more likely his publishing software, must now discover Alice’s Webmention endpoint. He does this by fetching Alice’s document via HTTP– if it has a Link header with a rel value of “webmention” that’s Alice’s webmention endpoint (this is how sending webmentions of non-HTML documents is supported). If there is no such header, and the document is HTML, Bob must look through the document itself for <link> or <a> elements with rel values of “webmention”– the first one in the document is Alice’s Webmention endpoint.

Bob then posts x-www-form-urlencoded source & target parameters to Alice’s Webmention endpoint (the target parameter being Alice’s post and the source being Bob’s mention thereof). Alice’s Webmention endpoint will typically do some initial validation of the request, and on success return either “201 Created” or “202 Accepted” to indicate that further processing will take place asynchronously (in the former case, the Location header in the response will include an URL at which the request processing status may be checked).

Asynchronously, Alice verifies the Webmention by issuing an HTTP GET on the source and verify that the source document contains an exact match for the URL given in the target parameter of the Webmention.

At this point, Alice may choose to publish content from Bob’s mention along with any other data it obtained (e.g. “Bob liked...”).

You can see an example of this in action (in conjunction with see POSSE) here.

2.1.4 POSSE ¶

“POSSE” stands for Publish on your Own Site, Syndicate Elsewhere” (or Everywhere). POSSE is less a formal protocol (like see Webmentions) than a feature you may want to build into your site.

The idea is that, after posting to one’s site, one then pushes copies of, or links to, that new post to assorted third-party sites (Twitter, for instance). This allows readers who prefer those sites to remain aware of your activity without leaving their prefered experience. This feature can be extended to collect responses to the POSSE’d content within the third-party sites (likes, replies and so forth) and feed that back to your site: a list of Tweets and/or Toots that have replied to you underneath your original post, for instance.

You can see an example of this in action (in conjunction with see Webmentions) here.

It stems from the Indieweb principle of owning your data rather than trusting it to a third-party site that may go away or simply decide to deny you access to your own content. It also stems from the hard fact that many of our friends & colleagues are still on such third-party sites such as Twitter or Mastodon: this is a way to connect your content thereto.

Implementations are generally bespoke; indie-org supports this for the site author via brid.gy (see see Supporting POSSE).

2.2.1 The Export Process ¶

This subsection outlines the generic Org Export process with an eye toward customizing it for our purposes. While we defer the fully-customized implementation to See Customizing the Publication Process, it is here that we introduce indie-org.sh, a live example of an Org-generated static site built on top of indie-org.

The Org Export publication process, in broad terms, looks like this:

The main entrypoint to the Org Export facility is See org-publish-all. org-publish-all refers to the variable See org-publish-project-alist, which describes our publication “project” to Org Mode. For each project (the root project can have sub-projects), Org Export will iterate over each file, copying it to its configured destination and, in general, transforming the contents in some way.

Of particular note is the info, or plist argument which is initialized at the beginning of the process of publishing each project (in org-publish-file) and is passed down through the entire call stack. It begins as the project plist (i.e. the project entry in org-publish-project-alist) and is augmented at each step. It is a property list that the different pieces of the publication pipeline use to communicate, and indie-org will use it as well.

The process of transforming the Org Mode document to another format is governed by the :publishing-function attribute for each project. It is invoked with the input filename, the publication directory, and the project plist.

indie-org.sh has two sub-projects salient to this discussion: “pages” and “posts”. The former contains a few top-level pages (home, about & so forth) whereas the latter contains all the site posts. Each employs a lambda as its publishing-function; the lambdas simply provide one more argument to the true publishing implementation, iosh/publish-page. The additional argument is the Org Export See Back-end in Adding Export Back-ends to use.

iosh/publish-page passes that back-end on org-publish-org-to and thence to org-export-to-file and org-export-as, which is where the real work begins. The function has a lot of functionality packed into its 171 lines; of note to us, it will:

1. Parse the buffer being exported into Org elements (see See org in Org Syntax)
2. Give the export backend the opportunity to filter the resulting parse tree; we’re not doing that so far in this example, but will take advantage of this below.
3. Transcodes the resulting parse tree to HTML
4. Extract the template from the backend & ask it to produce an HTML document from that HTML
5. Offers the backend the opportunity to “finalize” the resulting page; that is, to run arbitrary code after the HTML document has been produced.

We introduce an Emacs Lisp file, indie-org.sh.el to the project where we can setup this boilerplate & invoke org-publish-all:

(defun iosh/publish (prod)
  "Publish indie-org.sh to production if PROD is non-nil, locally else."
  ;; Define a derived backend to insert our template:
  (org-export-define-derived-backend
      'iosh/page-html
      'html
      :translate-alist '((template . iosh/page-html)))
  (org-export-define-derived-backend
      'iosh/post-html
      'html
      :translate-alist '((template . iosh/post-html)))
  ;; ...
  (let* ((publishing-root (concat (file-name-as-directory project-dir) "www"))
         (org-publish-project-alist
          `(("indie-org.sh" :components ("pages"))
            ("pages"
             :base-directory             ,(concat project-dir "pages")
             :publishing-directory       ,(concat project-dir "www")
             :publishing-function        ,(list
                                           (lambda (plist filename pub-dir)
                                             (iosh/publish-page plist filename pub-dir 'iosh/page-html)))
             :html-doctype               "html4-strict"
             ...
             )
            ("posts"
             :base-directory             ,(concat project-dir "posts")
             :publishing-directory       ,(concat project-dir "www/posts")
             :publishing-function        ,(list
                                           (lambda (plist filename pub-dir)
                                             (iosh/publish-page plist filename pub-dir 'iosh/post-html)))
             :html-doctype               "html4-strict"
             ...
             ))))
      (org-publish-all t)
      ;; ...
      ))

2.2.2 Can Org Mode Documents Exist Outside of Emacs? ¶

It has been suggested to the author that he has built far too much functionality in Emacs Lisp, and that if one is interested in putting a staticly generated site on the Indieweb, it is better implemented as a purpose-built program written in a general-purpose programming language.

Perhaps. However, if one is committed to writing in Org Mode, that avenue of attack on the problem leads us to an interesting question: how can we parse Org Mode files outside of Emacs?

Today, the reference implementation of an Org Mode parser is... Emacs Org Mode. The syntax is documented here and the implementation, in org-element.el is unusually readable, but it all exists within Emacs.

There are (numerous) parsers available, in a wide variety of languages, with various levels of fidelity to Org Mode claimed. There is an effort ongoing at the time of this writing on the emacs-orgmode list to standardize a set of test cases to make compliance more rigorous. Karl Voit independently gave a talk at EmacsConf 2021 proposing a specification which he called “Orgdown”.

So it may be that in the not too distant future, we will have Org Mode parsers in general-purpose languages which can be validated against some sort of specification, or at least a well-known test suite. However, and this has been discussed at length on the emacs-orgmode list, such a parser will either need to re-implement much of Emacs or be incomplete: how should such a parser handle Babel, for instance? Table formulas that invoke arbitrary Emacs Lisp functions? The author uses a custom Org Mode property, #+AUTODATE in his documents; if set to t, the #+DATE property will be updated every time the buffer is saved– how should that be handled?

Some argue that such a specification could only cover the “static” portions of an Org Mode document, but it’s not clear to the author exactly how that would work in the case of the “dynamic” portions generating document content.

The answer is not clear to me at this time. However, having implemented the functionaly to support putting my own sites on the Indieweb, it seemed reasonable to make that functionality available to others.

4.3.1 Using PGP ¶

You’ll need to generate a PGP keypair to identify your site, if you don’t already have a keypair you’d like to use. To generate a new keypair, say gpg --gen-key and follow the prompts. Think ahead of time about the e-mail you’d like to associate with the new keypair.

Once you’ve got your keypair, you can get its identifier by saying gpg --list-keys (the identifier is the long hexadecimal string listed for each public key on your keyring). Say gpg --output PATH-TO-YOUR-SOURCE/site-public.pgp --export KEYID to put a copy of your site’s public key somewhere in your site’s source project directory.

Add a link to your home page like:

<link rel="pgpkey" href="/site-public.pgp>

To test, go to https://indieauth.com and scroll down to the “Try It!” section.

Note that IndieAuth.com supports neither Twitter nor Mastodon for authentication purposes.

You should see a green button labelled “GPG” along with your domain & the name of your public key. Click that and you’ll be presented with a string to sign with your private key (demonstrating ownership of the private key). Copy the challenge text and say echo 'CHALLENGE-TEXT'|gpg -u KEYID --clearsign --armor. This should produce output something like:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

eyJ0eXAiOiJKV1Qi...
-----BEGIN PGP SIGNATURE-----

iQGzBAEBCAAdFiEEr2ZwJM99NLbLOVkMca2Hl/Lwd1cFAmQqHdQACgkQca2Hl/Lw
d1e1jwv/bpIrpr7+WNfD1xfiNzkq+PzbeeMT07B8kHo3ZKXJINB420jO3P+QqM8G
S1WQF2XyhnzxKmo/ySk54HOV5iWZ62uBHIrn/Nn6YUBvVQUB6CiF0zeCvKrbreW/
3omdbdLfCryPAMd120sQi8mQ5fDr798jq8Oq7QyIA4WusIh3ZesoDYboE4VJKryK
...
-----END PGP SIGNATURE-----

Paste the entire messages into the box.

4.3.2 Using Social Media Sites ¶

While IndieAuth does not support login via other sites, such as Twitter or Mastodon, IndieLogin does, and the services we’re going to be using use the latter, not the former.

In order to do this, the site author must mark-up their home page with links to their social media profiles with the rel="me" attribute, e.g.:

<link href="https://twiiter.com/jdoe" rel="me">

On each service listed, ensure there’s a link back to the site’s home page.

Supported services:

1. Twitter
2. Mastodon
3. Github
4. PGP key

4.4.1 Making & Recording Webmentions ¶

The first step in making a Webmention is to express that intent in our Org Mode source files. indie-org implements this by See Adding Hyperlink Types in Adding Hyperlink Types, Adding Hyperlink Types.

When the site author invokes indie-org-webmentions-enable four new link types will be defined:

1. mention
2. reply
3. like
4. repost

Each link type has a few properties, in particular :follow and :export. indie-org’s link types all implement follow in the same way as standard HTML links; it’s the export aspect in which they differ. While they will all render as standard HTML, they will take the opportunity to note each of them via indie-org-webmentions-record-webmention

The site author shall include their publication state’s indie-org-webmentions-made (see Publication Environments & State) in the project properties (under the name :indie-org/webmentions-made); at export time, the custom link types will, first, produce See Microformats appropriate to the sort of webmention being made, then invoke indie-org-webmentions-record-webmention to record them in the current publication environment’s state.

4.4.2 Sending Webmentions ¶

At this point, assuming the site author has See made & recorded one or more Webmentions, they presumably want to send them.

Despite the protocol’s simplicity, sending Webmentions is a non-trivial task. The sender needs to carry-out Webmention endpoint discovery on their targets (which itself involves fetching & parsing the resulting document). Then there’s the question of handling failures to send & re-trying. Finally, since Webmention processing is generally done asynchronously, the sender has to “check back” to get the final result.

Aaron Precki has built a service to handle this: Telegraph. Once you sign-up (using See IndieAuth, of course), you can send a Webmention with a single API call over HTTP.

indie-org simplifies this further with indie-org-webmentions-send a function that takes a source, target, and Telegraph API token and will send the Webmention (recall that the precise sort of Webmention is not encoded in the protocol; the receiver discovers it by parsing the sender’s post).

We can record sent Webmentions in indie-org-webmentions-sent, another attribute of indie-org-publication-state. The process of taking both & determining what Webmentions are to be sent is encapsulated in indie-org-webmentions-required.

So equipped, we can now code-up a function in our site’s Lisp that we can call post-publication that will handle sending all new Webmentions:

(defun my/send-webmentions (prod)
  "Send webmentions post-publication.
PROD shall be set to t for production and nil for staging."
  (indie-org-enable)
  (let* ((env (if prod :prod :staging))
         (all-pub-states
          (if (file-exists-p publication-state-file)
              (indie-org-state-read publication-state-file)))
         (publication-state
          (or (plist-get all-pub-states env)
              (indie-org-state-make)))
         (webmentions-made
          (or (indie-org-state-v2-webmentions-made publication-state)
              (indie-org-webmentions-make-made)))
         (webmentions-sent
          (or (indie-org-state-v2-webmentions-sent publication-state)
              (indie-org-webmentions-make-sent)))
         (webmentions-to-send
          (indie-org-webmentions-required webmentions-made webmentions-sent))
         (token
          (indie-org-read-token
           telegraph-io-token-file))
         (now (current-time)))
    (while webmentions-to-send
      (let* ((wm (car webmentions-to-send))
             (location
              (if prod
                  ;; The source is just the page key
                  ;; (e.g. "blog/foo.html")-- it is only here that we
                  ;; know we're sending this for prod, so prepend the
                  ;; "https://www.unwoundstack.com" here:
                  (let ((src (car wm))
                        (dst (cdr wm)))
                    (indie-org-webmentions-send
                     (cons (concat "https://www.coffeehouse-talker.net/" src) dst)
                     token))
                (message "Will send webmention: %s :=> %s" (car wm) (cdr wm))
                nil)))
        (indie-org-webmentions-record-sent
         wm
         now
         webmentions-sent
         location))
      (setq webmentions-to-send (cdr webmentions-to-send)))
    (setf (indie-org-state-v2-webmentions-sent publication-state) webmentions-sent)
    (plist-put all-pub-states env publication-state)
    (indie-org-state-write all-pub-states publication-state-file)))

4.4.3 Receiving Webmentions ¶

Receiving Webmentions may seem like a particularly daunting problem for the staticly generated site owner. In order to receive them, we need a server capable of handling the HTTP posts and implement the server-side of the protocol (Webmention validation, asynchronous processing, possibly another endpoint at which senders can check their results).

It turns out, there’s a service for that: webmention.io. Once you sign-up via See IndieAuth, advertise your Webmention endpoint like so:

<link rel="webmention" href="https://webmention.io/indie-org.sh/webmention" />
<link rel="pingback" href="https://webmention.io/indie-org.sh/xmlrpc" />

Webmention senders will now POST to webmention.io who will store them on your behalf. You can retrieve them via API call to webmention.io.

indie-org handles this for you via indie-org-webmentions-check. This function takes the domain name for which you are checking Webmentions, your webmtion.io API token, and an indie-org-webmentions-received struct (another attribute of indie-org-publication-state). indie-org-webmentions-check will update the struct with any new Webmentions made.

So armed, we can add a method to our site’s Lisp, meant to be invoked periodically, for checking Webmentions made to our site:

(defun my/check-webmentions ()
  "Check for new webmentions; update publication state."
  (indie-org-enable)
  (let* ((env :prod) ;; no staging support ATM
         (publication-state
          (if (file-exists-p publication-state-file)
              (indie-org-state-read publication-state-file)))
         (state-for-env
          (or (plist-get publication-state env) (indie-org-state-make)))
         (webmentions-received
          (or (indie-org-state-v2-webmentions-received state-for-env)
              (indie-org-webmentions-make-received)))
         (token
          (indie-org-read-token
           webmentions-io-token-file)))
    (indie-org-webmentions-check "indie-org.sh" token webmentions-received)
    (setf (indie-org-state-v2-webmentions-received state-for-env) webmentions-received)
    (plist-put publication-state env state-for-env)
    (indie-org-state-write publication-state publication-state-file)))

The site owner will need to arrange for this method to be invoked on a regular basis, perhaps as a cron job. If the owner wishes to display mentions as part of their page template(s), they will need to re-publish on receipt of new Webmentions.

4.5.1 Setting-up POSSE to Twitter & Mastodon ¶

4.5.2 Handling Backfeed ¶

4.5.3 Controlling Content ¶