Better Integration with Wiki style syntax

Hi

Even if the Footnotes module itself doesn't mind you using it together with some Wiki filter, in practice you should just use the type of Footnotes filter that matches the general markup style you are using. So, the correct solution would be to develop a third Footnotes style, compatible with wiki style. (Both for the Footnotes markup you write and what is output by the filter.) This is actually fairly simple to do, if you know basic php and preg_replace() you can start with copy pasting some code in the following 3 methods:

/**
* Implementation of hook_filter_tips().
*
* This hook allows filters to provide help text to users during the content
* editing process. Short tips are provided on the content editing screen, while
* long tips are provided on a separate linked page. Short tips are optional,
* but long tips are highly recommended.
*/
function footnotes_filter_tips($delta, $format, $long = FALSE) {
switch ($delta) {
case 0:
if ($long) {
return t('You can insert footnotes directly into texts with <fn>This text becomes a footnote.</fn>. This will be replaced with a running number (the footnote reference) and the text within the <fn> tags will be moved to the bottom of the page (the footnote).');
}
else {
return t('Use <fn>...</fn> to insert automatically numbered footnotes.');
}
break;

case 1:
if ($long) {
return t('You can insert footnotes directly into texts with [# ...]. This will be replaced with a running number (the footnote reference) and the text within the [# ...] tags will be moved to the bottom of the page (the footnote). This filter outputs footnotes in Textile format. You should use it together and before the Textile filter.');
}
else {
return t('Use [# ...] to insert automatically numbered footnotes. Textile variant.');
}
break;
}
}

/**
* Implementation of hook_filter().
*
* The bulk of filtering work is done here. This hook is quite complicated, so
* we'll discuss each operation it defines.
*/
function footnotes_filter($op, $delta = 0, $format = -1, $text = '') {
// The "list" operation provides the module an opportunity to declare both how
// many filters it defines and a human-readable name for each filter. Note that
// the returned name should be passed through t() for translation.
if ($op == 'list') {
return array(
0 => t('Footnotes <fn>'),
1 => t('Footnotes Textile style'));
}

// All operations besides "list" provide a $delta argument so we know which
// filter they refer to. We'll switch on that argument now so that we can
// discuss each filter in turn.
switch ($delta) {

// First is the html footnotes filter
case 0:

switch ($op) {
// This description is shown in the administrative interface, unlike the
// filter tips which are shown in the content editing interface.
case 'description':
return t('Use <fn>...</fn> to insert automatically numbered footnotes.');

// We don't need the "prepare" operation for this filter, but it's required
// to at least return the input text as-is.
//TODO: May need to escape if we use HTML filter too, but Footnotes could be first
case 'prepare':
return $text;

// The actual filtering is performed here. The supplied text should be
// returned, once any necessary substitutions have taken place.
case 'process':
$text = preg_replace_callback('|(.*?)|s', '_footnotes_replace_callback', $text);

//Replace tag with the list of footnotes.
//If tag is not present, by default add the footnotes at the end.
//Thanks to acp on drupal.org for this idea. see http://drupal.org/node/87226
$footer = '';
$footer = _footnotes_replace_callback(NULL, 'output footer');
if( preg_match( '//', $text ) > 0 ) {
$text = preg_replace('//', $footer, $text, 1);
return $text;
}
else {
return $text . "\n\n" . $footer;
}
}
break;

// Textile version.
case 1:

switch ($op) {
// This description is shown in the administrative interface, unlike the
// filter tips which are shown in the content editing interface.
case 'description':
return t('Use [# ...] to insert automatically numbered footnotes in Textile markup.');

// We don't need the "prepare" operation for this filter, but it's required
// to at least return the input text as-is.
case 'prepare':
return $text;

// The actual filtering is performed here. The supplied text should be
// returned, once any necessary substitutions have taken place.
case 'process':
$text = preg_replace_callback('|\[# (.*?)\]|s', '_footnotes_replace_callback_textile', $text);

//Replace Textile tag "footnotes." with the list of footnotes.
//If tag is not present, by default add the footnotes at the end.
//Thanks to acp on drupal.org for this idea. see http://drupal.org/node/87226
$footer = '';
$footer = _footnotes_replace_callback_textile(NULL, 'output footer');
if( preg_match( '/\n *footnotes\. *(\n|$)/', $text ) > 0 ) {
$text = preg_replace('/\n *footnotes\. *(\n|$)/', "\n$footer\n", $text, 1);
return $text;
}
else {
return $text . "\n\n" . $footer;
}
}
break;
}
}

/**
* Helper function called from preg_replace_callback() above
*
* Uses static vars to temporarily store footnotes found.
* In my understanding, this is not threadsafe?!
*/
function _footnotes_replace_callback( $matches, $op = '' ) {
static $n = 0;
static $store_matches = array();
$str = '';

if( $op == 'output footer' ) {
if( $n > 0 ) {
$str = '

';
for( $m = 1; $m <= $n; $m++ ){
$str .= '

1. ' . $store_matches[ $m - 1 ] . "

\n\n";
}
$str .= '

';
}
$n = 0;
$store_matches = array();
return $str;
}

//default op: act as called by preg_replace_callback()
array_push( $store_matches, $matches[1] );
$n++;
$allowed_tags = array();
$title = filter_xss($matches[1], $allowed_tags);
//html attribute cannot contain quotes
$title = str_replace('"', """, $title);
//remove newlines. Browsers don't support them anyway and they'll confuse line break converter in filter.module
$title = str_replace("\n", " ", $title);
$title = str_replace("\r", "", $title);
return '^{' . $n . '}';
}

And as always, if you provide a patch, we will add it to the next release!

I've reviewed this and tried to see how it could be implemented.

Review

The PEAR Wiki Filter actually supports several markup styles: Mediawiki, TikiWiki, DokuWiki, Creole and BBCode.

I will skip Creole, as I understand it is not widespread. BBCode is a simple markup system used mainly in forums and I don't see a critical need for footnotes in it either. This leaves us with 3 wiki markup styles.

The Dokuwiki syntax defines markup for footnotes (http://wiki.splitbrain.org/wiki:syntax#footnotes) as does MediaWiki (http://meta.wikimedia.org/wiki/Help:Footnotes). TikiWiki doesn't seem to support footnotes, but it does have an api for arbitrary plugins (http://tikiwiki.org/tiki-index.php?page=PluginsList&bl) , which would be the right way to implement this for TikiWiki.

The problem, as far as I can see, is that none of the implementations in PEAR Text_Wiki_* actually support footnotes. (They are independent parsers that don't support the full syntaxes and all are also quite old it seems.)

Solutions

1) The right thing to do is obviously to go to the PEAR modules in question and add footnotes support to at least the MediaWiki and DocuWiki modules, in their respective syntaxes.

2) However, I do admit that at least the Textile variant in the Footnotes filter came to be simply because "Uh, I just want a simple hack that will support footnotes, I don't want to touch the Textile code itself." Therefore, I am sympathetic to the idea of Footnotes module supporting more markup styles.

Problems

The current style of Footnotes module is that the HTML variant reads a HTMLesque style of footnotes and outputs HTML. OTOH the Textile variant reads a footnote style similar to other Textile commands and outputs valid Textile syntax. (Which the Textile filter then makes into HTML.) In keeping with this strategy we should now implement 3 new variants for each of the MediaWiki, DocuWiki and TikiWiki syntaxes.

There is one problem with the above approach, that in creating the list of footnotes we are limited by the syntax of the variant we are using. Already now the Textile variant is not as cool as the HTML variant (the reference is not a link to the footnote, it is not as rich in CSS...). In particular, I don't see any of the above Wiki languages supporting links within the page (ie between the number in the main text and the footnote at the bottom). It seems that at best the Wiki languages support linking to headers only.

A smaller problem is of course just the need to support an ever growing list of syntaxes. This is not critical, but a drawback nevertheless.

Solution

It could be a better idea to restructure the Footnotes module as follows:

1. You would always use the html variant, regardless of what markup syntax you text is using.
2. Footnotes filter should run after your main markup syntax filter (Textile, MediaWiki, DokuWiki, etc...). (The current version is designed to run first.)
3. To enable this, footnotes_filter() must support the "prepare" operation.
4. Footnotes should still run before HTML filter, HTML corrector and other "cleanup" filters. (Because as long as the footnotes are not "in the right place", more complex footnotes will make the markup invalid html and confuse these filters.
5. Textile variant of Footnotes would be deprecated and removed after 2 or more years.

Example

Imagine some text that originally looks like this:

This is some example text in a Wiki Format. I can use *bold* and /italic/ and whatever, or [Links to other documents]. However, I will create footnotes<fn>With the same syntax as footnotes are always created in Drupal.</fn> Please note that html <b>may not be supported</b> by the Wiki filter!

When user posts this text, the filters kicks in. First the Footnotes prepare operation shields the <fn> tags. (See the time tag filter in http://api.drupal.org/api/HEAD/function/filter_example_filter)

This is some example text in a Wiki Format. I can use *bold* and /italic/ and whatever, or [Links to other documents]. However, I will create footnotes \xFEfn\xFFWith the same syntax as footnotes are always created in Drupal.\xFE/fn\xFF Please note that html <b>may not be supported</b> by the Wiki filter!

Note that the codes \xFE and \xFF are not really text, they are Unicode values that in real life are invisible.

Then the Wiki filter does it's job:

This is some example text in a Wiki Format. I can use <b>bold</b> and <i>italic</i> and whatever, or <a href="Links to other documents">Links to other documents</a>. However, I will create footnotes \xFEfn\xFFWith the same syntax as footnotes are always created in Drupal.\xFE/fn\xFF Please note that html may not be supported by the Wiki filter!

Note that a typical wiki filter may (or may not) strip away html tags (like <b>) for security reasons. That's why <fn> had to be prepared.

Now it is time to do the footnotes:

This is some example text in a Wiki Format. I can use <b>bold</b> and <i>italic</i> and whatever, or <a href="Links to other documents">Links to other documents</a>. However, I will create footnotes<sup><a href="#footnote1">1</a></sup>Please note that html may not be supported by the Wiki filter!

<dl class="footnotes">
<dt><a class="footnote" name="footnote1" href="#footnoteref1">1.</a></dt>
<dd>With the same syntax as footnotes are always created in Drupal.</dd>
</dl>

And after this HTML filter and HTML corrector may still do their jobs

PS

I've used some days of my vacation for maintenance of the Footnotes module. (And it's been great and I wish I could spend more time on it!) However, my vacation is now nearing its end so I just want to stress that the above review doesn't mean that I would actually have time to implement this any time soon. But this will probably be the next thing I implement whenever I have time to do that.

Just to make things more difficult, the people who developed that wiki module are actually recommending that you go with the new flexifilters module, and they don't intend to upgrade to 6.x

http://drupal.org/project/flexifilter

HTH,

Hi Wayland and thanks for keeping me updated.

While there has been no progress on this issue, I've kept my mind on it for a year now. I've tried to come up with various ways to best integrate with various wiki's. My conclusion is that only 2 options are good:

* Footnotes feature should be integrated with the various wiki-filters themselves. (As is the case with the real Mediawiki.) This is the best alternative as it allows for the best integration. However, this is obviously none of my business as the Footnotes maintainter.
* Implement a new Wiki-style Footnotes filter, just as we now have two, the Html and Textile style filters. This filter will accept footnotes with markup that fits into the wiki-style overall markup and outputs markup that is not html but rather something that one or more wiki filters will understand as input.

I've thought of various schemes to convert wiki-style input directly into html output, but this leads to all kinds of problems as to ordering of filters, so I am not considering them anymore.

So as a summary, I still intend to implement the second bullet point one day. As soon as #161217: URL filter breaks generated href tags is closed I guess.

I tried to Flexifilter, but it seems really really rudimentary and not ready for real use. It didn't support creating a simple bulleted list, and for some funny reason all links point to en.wikipedia.org.

I tried with the MediaWiki version of pearwiki_filter, which seems more usable. In fact you can use the current HTML filter just fine, the resulting HTML is passed throught the MediaWiki filter.

Re-reading the original question, it seems some of the new features are what asbdpl was asking for and by using the newest version of Footnotes, all should work fine. (Use the [fn]...[/fn] or ... version, not the Textile version.)

In particular, to get the URLs into links, replace Drupal's own URL filter with the "Better URL filter" that comes with Footnotes 6.x-2.x-dev version now.

Is there any work around for footnotes with mediawiki input format for 5.7 ? because I am certainly not able to use it

Oh?

The http://ftp.drupal.org/files/projects/footnotes-6.x-2.x-dev.tar.gz version should be usable on 5.7, there is nothing 6.x specific in it. The [fn]...[/fn] style footnotes filter should work just fine with Mediawiki. If your experience is the opposite, please explain here.

hingo, you rocks! Working like a charm..

Heh. Thanks :-)