1 The u32 page is excellent, but I wonder if documenting the procedure here
2 would be worthwhile. Who knows, the remote site might disappear. But also
3 there are some variations on the approach that might be useful:
5 * using a python script and the dom library to extract the page names from
6 Special:Allpages (such as
7 <http://www.staff.ncl.ac.uk/jon.dowland/unix/docs/get_pagenames.py>)
8 * Or, querying the mysql back-end to get the names
9 * using WWW::MediaWiki for importing/exporting pages from the wiki, instead
12 Also, some detail on converting mediawiki transclusion to ikiwiki inlines...
16 > "Who knows, the remote site might disappear.". Right now, it appears to
17 > have done just that. -- [[users/Jon]]
19 I have manage to recover most of the site using the Internet Archive. What
20 I was unable to retrieve I have rewritten. You can find a copy of the code
21 at <http://github.com/mithro/media2iki>
24 The iki-fast-load ruby script from the u32 page is given below:
28 # This script is called on the final sorted, de-spammed revision
31 # It doesn't currently check for no-op revisions... I believe
32 # that git-fast-load will dutifully load them even though nothing
33 # happened. I don't care to solve this by adding a file cache
34 # to this script. You can run iki-diff-next.rb to highlight any
35 # empty revisions that need to be removed.
37 # This turns each node into an equivalent file.
38 # It does not convert spaces to underscores in file names.
39 # This would break wikilinks.
40 # I suppose you could fix this with mod_speling or mod_rewrite.
42 # It replaces nodes in the Image: namespace with the files themselves.
46 require 'node-callback'
51 # pipe is the stream to receive the git-fast-import commands
52 # putfrom is true if this branch has existing commits on it, false if not.
53 def format_git_commit(pipe, f)
54 # Need to escape backslashes and double-quotes for git?
55 # No, git breaks when I do this.
56 # For the filename "path with \\", git sez: bad default revision 'HEAD'
57 # filename = '"' + filename.gsub('\\', '\\\\\\\\').gsub('"', '\\"') + '"'
59 # In the calls below, length must be the size in bytes!!
60 # TODO: I haven't figured out how this works in the land of UTF8 and Ruby 1.9.
61 pipe.puts "commit #{f.branch}"
62 pipe.puts "committer #{f.username} <#{f.email}> #{f.timestamp.rfc2822}"
63 pipe.puts "data #{f.message.length}\n#{f.message}\n"
64 pipe.puts "from #{f.branch}^0" if f.putfrom
65 pipe.puts "M 644 inline #{f.filename}"
66 pipe.puts "data #{f.content.length}\n#{f.content}\n"
70 > Would be nice to know where you could get "node-callbacks"... this thing is useless without it. --[[users/simonraven]]
73 Mediawiki.pm - A plugin which supports mediawiki format.
76 # By Scott Bronson. Licensed under the GPLv2+ License.
77 # Extends Ikiwiki to be able to handle Mediawiki markup.
79 # To use the Mediawiki Plugin:
80 # - Install Text::MediawikiFormat
81 # - Turn of prefix_directives in your setup file.
82 # (TODO: we probably don't need to do this anymore?)
83 # prefix_directives => 1,
84 # - Add this plugin on Ikiwiki's path (perl -V, look for @INC)
85 # cp mediawiki.pm something/IkiWiki/Plugin
86 # - And enable it in your setup file
87 # add_plugins => [qw{mediawiki}],
88 # - Finally, turn off the link plugin in setup (this is important)
89 # disable_plugins => [qw{link}],
90 # - Rebuild everything (actually, this should be automatic right?)
91 # - Now all files with a .mediawiki extension should be rendered properly.
94 package IkiWiki::Plugin::mediawiki;
102 # This is a gross hack... We disable the link plugin so that our
103 # linkify routine is always called. Then we call the link plugin
104 # directly for all non-mediawiki pages. Ouch... Hopefully Ikiwiki
105 # will be updated soon to support multiple link plugins.
106 require IkiWiki::Plugin::link;
108 # Even if T:MwF is not installed, we can still handle all the linking.
109 # The user will just see Mediawiki markup rather than formatted markup.
110 eval q{use Text::MediawikiFormat ()};
111 my $markup_disabled = $@;
113 # Work around a UTF8 bug in Text::MediawikiFormat
114 # http://rt.cpan.org/Public/Bug/Display.html?id=26880
115 unless($markup_disabled) {
118 *{'Text::MediawikiFormat::uri_escape'} = \&URI::Escape::uri_escape_utf8;
121 my %metaheaders; # keeps track of redirects for pagetemplate.
122 my %tags; # keeps track of tags for pagetemplate.
126 hook(type => "checkconfig", id => "mediawiki", call => \&checkconfig);
127 hook(type => "scan", id => "mediawiki", call => \&scan);
128 hook(type => "linkify", id => "mediawiki", call => \&linkify);
129 hook(type => "htmlize", id => "mediawiki", call => \&htmlize);
130 hook(type => "pagetemplate", id => "mediawiki", call => \&pagetemplate);
136 return IkiWiki::Plugin::link::checkconfig(@_);
140 my $link_regexp = qr{
141 \[\[(?=[^!]) # beginning of link
142 ([^\n\r\]#|<>]+) # 1: page to link to
144 \# # '#', beginning of anchor
145 ([^|\]]+) # 2: anchor text
150 ([^\]\|]*) # 3: link text
153 ([a-zA-Z]*) # optional trailing alphas
157 # Convert spaces in the passed-in string into underscores.
158 # If passed in undef, returns undef without throwing errors.
162 $var =~ tr{ }{_} if $var;
167 # Underscorize, strip leading and trailing space, and scrunch
168 # multiple runs of spaces into one underscore.
173 $var =~ s/^\s+|\s+$//g; # strip leading and trailing space
174 $var =~ s/\s+/ /g; # squash multiple spaces to one
180 # Translates Mediawiki paths into Ikiwiki paths.
181 # It needs to be pretty careful because Mediawiki and Ikiwiki handle
182 # relative vs. absolute exactly opposite from each other.
186 my $path = scrunch(shift);
188 # always start from root unless we're doing relative shenanigans.
189 $page = "/" unless $path =~ /^(?:\/|\.\.)/;
192 for(split(/\//, "$page/$path")) {
196 push @result, $_ if $_ ne "";
200 # temporary hack working around http://ikiwiki.info/bugs/Can__39__t_create_root_page/index.html?updated
201 # put this back the way it was once this bug is fixed upstream.
202 # This is actually a major problem because now Mediawiki pages can't link from /Git/git-svn to /git-svn. And upstream appears to be uninterested in fixing this bug. :(
203 # return "/" . join("/", @result);
204 return join("/", @result);
208 # Figures out the human-readable text for a wikilink
211 my($page, $inlink, $anchor, $title, $trailing) = @_;
212 my $link = translate_path($page,$inlink);
214 # translate_path always produces an absolute link.
215 # get rid of the leading slash before we display this link.
220 $out = IkiWiki::pagetitle($title);
222 $link = $inlink if $inlink =~ /^\s*\//;
223 $out = $anchor ? "$link#$anchor" : $link;
224 if(defined $title && $title eq "") {
225 # a bare pipe appeared in the link...
226 # user wants to strip namespace and trailing parens.
227 $out =~ s/^[A-Za-z0-9_-]*://;
228 $out =~ s/\s*\(.*\)\s*$//;
230 # A trailing slash suppresses the leading slash
231 $out =~ s#^/(.*)/$#$1#;
233 $out .= $trailing if defined $trailing;
242 if (exists $config{tagbase} && defined $config{tagbase}) {
243 $tag=$config{tagbase}."/".$tag;
250 # Pass a URL and optional text associated with it. This call turns
251 # it into fully-formatted HTML the same way Mediawiki would.
252 # Counter is used to number untitled links sequentially on the page.
253 # It should be set to 1 when you start parsing a new page. This call
254 # increments it automatically.
255 sub generate_external_link
261 # Mediawiki trims off trailing commas.
262 # And apparently it does entity substitution first.
263 # Since we can't, we'll fake it.
265 # trim any leading and trailing whitespace
266 $url =~ s/^\s+|\s+$//g;
268 # url properly terminates on > but must special-case >
270 $url =~ s{(\&(?:gt|lt)\;.*)$}{ $trailer = $1, ''; }eg;
272 # Trim some potential trailing chars, put them outside the link.
274 $url =~ s{([,)]+)$}{ $tmptrail .= $1, ''; }eg;
275 $trailer = $tmptrail . $trailer;
280 $text = "[$$counter]";
283 $text =~ s/^\s+|\s+$//g;
289 return "<a href='$url' title='$title'>$text</a>$trailer";
293 # Called to handle bookmarks like \[[#heading]] or <span class="createlink"><a href="http://u32.net/cgi-bin/ikiwiki.cgi?page=%20text%20&from=Mediawiki_Plugin%2Fmediawiki&do=create" rel="nofollow">?</a>#a</span>
294 sub generate_fragment_link
301 $url = scrunch($url);
303 if(defined($text) && $text ne "") {
304 $text = scrunch($text);
309 $url = underscorize($url);
311 # For some reason Mediawiki puts blank titles on all its fragment links.
312 # I don't see why we would duplicate that behavior here.
313 return "<a href='$url'>$text</a>";
317 sub generate_internal_link
319 my($page, $inlink, $anchor, $title, $trailing, $proc) = @_;
321 # Ikiwiki's link link plugin wrecks this line when displaying on the site.
322 # Until the code highlighter plugin can turn off link finding,
323 # always escape double brackets in double quotes: \[[
324 if($inlink eq '..') {
325 # Mediawiki doesn't touch links like \[[..#hi|ho]].
326 return "\[[" . $inlink . ($anchor?"#$anchor":"") .
327 ($title?"|$title":"") . "]]" . $trailing;
330 my($linkpage, $linktext);
331 if($inlink =~ /^ (:?) \s* Category (\s* \: \s*) ([^\]]*) $/x) {
332 # Handle category links
335 $linkpage = IkiWiki::linkpage(translate_path($page, $inlink));
337 # Produce a link but don't add this page to the given category.
338 $linkpage = tagpage($linkpage);
339 $linktext = ($title ? '' : "Category$sep") .
340 linktext($page, $inlink, $anchor, $title, $trailing);
341 $tags{$page}{$linkpage} = 1;
343 # Add this page to the given category but don't produce a link.
344 $tags{$page}{$linkpage} = 1;
345 &$proc(tagpage($linkpage), $linktext, $anchor);
349 # It's just a regular link
350 $linkpage = IkiWiki::linkpage(translate_path($page, $inlink));
351 $linktext = linktext($page, $inlink, $anchor, $title, $trailing);
354 return &$proc($linkpage, $linktext, $anchor);
362 my $page=$params{page};
363 my $destpage=$params{destpage};
364 my $content=$params{content};
366 return "" if $page ne $destpage;
368 if($content !~ /^ \s* \#REDIRECT \s* \[\[ ( [^\]]+ ) \]\]/x) {
369 # this page isn't a redirect, render it normally.
373 # The rest of this function is copied from the redir clause
374 # in meta::preprocess and actually handles the redirect.
377 $value =~ s/^\s+|\s+$//g;
380 if ($value !~ /^\w+:\/\//) {
382 my ($redir_page, $redir_anchor) = split /\#/, $value;
384 add_depends($page, $redir_page);
385 my $link=bestlink($page, underscorize(translate_path($page,$redir_page)));
386 if (! length $link) {
387 return "<b>Redirect Error:</b> <nowiki>\[[$redir_page]] not found.</nowiki>";
390 $value=urlto($link, $page);
391 $value.='#'.$redir_anchor if defined $redir_anchor;
394 # redir cycle detection
395 $pagestate{$page}{mediawiki}{redir}=$link;
398 while (exists $pagestate{$at}{mediawiki}{redir}) {
400 return "<b>Redirect Error:</b> cycle found on <nowiki>\[[$at]]</nowiki>";
403 $at=$pagestate{$at}{mediawiki}{redir};
406 # it's an external link
407 $value = encode_entities($value);
410 my $redir="<meta http-equiv=\"refresh\" content=\"0; URL=$value\" />";
411 $redir=scrub($redir) if !$safe;
412 push @{$metaheaders{$page}}, $redir;
414 return "Redirecting to $value ...";
418 # Feed this routine a string containing <nowiki>...</nowiki> sections,
419 # this routine calls your callback for every section not within nowikis,
420 # collecting its return values and returning the rewritten string.
429 for(split(/(<nowiki[^>]*>.*?<\/nowiki\s*>)/s, $content)) {
430 $result .= ($state ? $_ : &$proc($_));
438 # Converts all links in the page, wiki and otherwise.
443 my $page=$params{page};
444 my $destpage=$params{destpage};
445 my $content=$params{content};
447 my $file=$pagesources{$page};
448 my $type=pagetype($file);
451 if($type ne 'mediawiki') {
452 return IkiWiki::Plugin::link::linkify(@_);
455 my $redir = check_redirect(%params);
456 return $redir if defined $redir;
458 # this code was copied from MediawikiFormat.pm.
459 # Heavily changed because MF.pm screws up escaping when it does
460 # this awful hack: $uricCheat =~ tr/://d;
461 my $schemas = [qw(http https ftp mailto gopher)];
462 my $re = join "|", map {qr/\Q$_\E/} @$schemas;
463 my $schemes = qr/(?:$re)/;
464 # And this is copied from URI:
465 my $reserved = q(;/?@&=+$,); # NOTE: no colon or [] !
466 my $uric = quotemeta($reserved) . $URI::unreserved . "%#";
468 my $result = skip_nowiki($content, sub {
472 #s/<(a[\s>\/])/<$1/ig;
473 # Disabled because this appears to screw up the aggregate plugin.
474 # I guess we'll rely on Iki to post-sanitize this sort of stuff.
476 # Replace external links, http://blah or [http://blah]
477 s{\b($schemes:[$uric][:$uric]+)|\[($schemes:[$uric][:$uric]+)([^\]]*?)\]}{
478 generate_external_link($1||$2, $3, \$counter)
481 # Handle links that only contain fragments.
482 s{ \[\[ \s* (\#[^|\]'"<>&;]+) (?:\| ([^\]'"<>&;]*))? \]\] }{
483 generate_fragment_link($1, $2)
486 # Match all internal links
488 generate_internal_link($page, $1, $2, $3, $4, sub {
489 my($linkpage, $linktext, $anchor) = @_;
490 return htmllink($page, $destpage, $linkpage,
491 linktext => $linktext,
492 anchor => underscorize(scrunch($anchor)));
503 # Find all WikiLinks in the page.
507 my $page=$params{page};
508 my $content=$params{content};
510 my $file=$pagesources{$page};
511 my $type=pagetype($file);
513 if($type ne 'mediawiki') {
514 return IkiWiki::Plugin::link::scan(@_);
517 skip_nowiki($content, sub {
519 while(/$link_regexp/g) {
520 generate_internal_link($page, $1, '', '', '', sub {
521 my($linkpage, $linktext, $anchor) = @_;
522 push @{$links{$page}}, $linkpage;
531 # Convert the page to HTML.
535 my $page = $params{page};
536 my $content = $params{content};
539 return $content if $markup_disabled;
541 # Do a little preprocessing to babysit Text::MediawikiFormat
542 # If a line begins with tabs, T:MwF won't convert it into preformatted blocks.
543 $content =~ s/^\t/ /mg;
545 my $ret = Text::MediawikiFormat::format($content, {
547 allowed_tags => [#HTML
548 # MediawikiFormat default
549 qw(b big blockquote br caption center cite code dd
550 div dl dt em font h1 h2 h3 h4 h5 h6 hr i li ol p
551 pre rb rp rt ruby s samp small strike strong sub
552 sup table td th tr tt u ul var),
556 qw(del ins), # These should have been added all along.
557 qw(span), # Mediawiki allows span but that's rather scary...?
558 qw(a), # this is unfortunate; should handle links after rendering the page.
562 qw(title align lang dir width height bgcolor),
565 qw(cite), # BLOCKQUOTE, Q
566 qw(size face color), # FONT
567 # For various lists, mostly deprecated but safe
568 qw(type start value compact),
570 qw(summary width border frame rules cellspacing
571 cellpadding valign char charoff colgroup col
572 span abbr axis headers scope rowspan colspan),
573 qw(id class name style), # For CSS
588 # This is only needed to support the check_redirect call.
592 my $page = $params{page};
593 my $destpage = $params{destpage};
594 my $template = $params{template};
596 # handle metaheaders for redirects
597 if (exists $metaheaders{$page} && $template->query(name => "meta")) {
598 # avoid duplicate meta lines
600 $template->param(meta => join("\n", grep { (! $seen{$_}) && ($seen{$_}=1) } @{$metaheaders{$page}}));
603 $template->param(tags => [
605 link => htmllink($page, $destpage, tagpage($_), rel => "tag")
606 }, sort keys %{$tags{$page}}
607 ]) if exists $tags{$page} && %{$tags{$page}} && $template->query(name => "tags");
609 # It's an rss/atom template. Add any categories.
610 if ($template->query(name => "categories")) {
611 if (exists $tags{$page} && %{$tags{$page}}) {
612 $template->param(categories => [map { category => $_ },
613 sort keys %{$tags{$page}}]);
620 Hello. Got ikiwiki running and I'm planning to convert my personal
621 Mediawiki wiki to ikiwiki so I can take offline copies around. If anyone
622 has an old copy of the instructions, or any advice on where to start I'd be
623 glad to hear it. Otherwise I'm just going to chronicle my journey on the
624 page.--[[users/Chadius]]
626 > Today I saw that someone is working to import wikipedia into git.
627 > <http://www.gossamer-threads.com/lists/wiki/foundation/181163>
628 > Since wikipedia uses mediawiki, perhaps his importer will work
629 > on mediawiki in general. It seems to produce output that could be
630 > used by the [[plugins/contrib/mediawiki]] plugin, if the filenames
631 > were fixed to use the right extension. --[[Joey]]