We had a situation last week where someone had entered a broken <iframe> tag in a job description and our cleanup code didn’t properly remove it. This caused the text after the <iframe> to render as escaped HTML.

We needed to prefilter the HTML and just remove the <iframe>s. The most difficult part of this was figuring out what HTML::TreeBuilder was emitting and what I needed to do with it to do the cleanup. It was obvious that this would have to be recursive, since HTML is recursive (there could be nested, or multiple uncosed iframes!) and several tries at it failed until I finally dumped out the data structure in the debugger and spotted that HTML::TreeBuilder was adding “implicit” nodes. These essentially help it do bookkeeping, but don’t contain anything that has to be re-examined to properly do the cleanup. Worse, the first node contains all th text for the current level, so recursing on them was leading me off into infinite depths, as I kept looking for iframes in the content of the leftmost node, finding them, and uselessly recursing again on the same HTML.

The other interesting twist is that once I dropped the implicit nodes with a grep, I still needed to handle the HTML in the non-implicit nodes two different ways: if it had one or more iframe tags, then I needed to use the content method to take the node apart and process the pieces. There might be one or more non-iframes there, which end up getting returned untouched via as_HTML. If there are iframes, the recursion un-nests them and lets us clean up individual subtrees.

Lastly, any text returned from content comes back as an array of strings, so I needed to check for that case and recurse on all the items in the array to be sure I’ve filtered everything properly. My initial case checks for the trivial “no input so no output”, and “not a reference” to handle the starting string.

We do end up doing multiple invocations of HTML::TreeBuilder on the text as we recurse, but we don’t recurse at all unless there’s an iframe, and it’s unusual to have more than one.

Here’s the code:

+sub _filter_iframe_content {
  my($input) = @_;
  return '' unless $input;

  my $root;
  # We've received a string. Build the tree.
  if (!ref $input) {
    # Build a tree to process recursively.
    $root = HTML::TreeBuilder->new_from_content($input);
    # There are no iframe tags, so we're done with this segment of the HTML.
    return $input unless $root->look_down(_tag=>'iframe');
  } elsif (ref $input eq 'ARRAY') {
    # We got multiple strings from a content call; handle each one in order, and
    # return them, concatenated, to finish them up.
    return join '', map { _filter_iframe_content($_) } @$input;
  } else {
    # The input was a node, so make that the root of the (sub)tree we're processing.
    $root = $input;
  }

  # The 'implicit' nodes contain the wrapping HTML created by
  # TreeBuilder. Discard that.
  my @descendants = grep { ! $_->implicit } $root->descendants;

  # If there is not an iframe below the content of the node, return
  # it as HTML. Else recurse on the content to filter it.
  my @results;
  for my $node (@descendants) {
    # Is there an iframe in here?
    my $tree = HTML::TreeBuilder->new_from_content($node->as_HTML);
    if ($tree->look_down(_tag=>'iframe')) {
      # Yes. Recurse on the node, taking it apart.
      push @results, _filter_iframe_content($node->content);
    } else {
      # No, just return the whole thing as HTML, and we're done with this subtree.
      push @results, $node->as_HTML;
    }
  }
  return join '', @results;
}

I lost an important VCVRack patch a couple days before Mountain Skies 2019. It was based on a patch I’d gotten from patchstorage.com, but I couldn’t remember which patch it was. I tried paging through the patches on the infinite scroll, but it wasn’t helping me much. I knew the patch had Clocked and the Impromptu 16-step sequencer, but I couldn’t remember anything else about it after seriously altering it for my needs.

I decided the only option was going to have to be automated if I was going to find the base patch again in time to recreate my performance patch. I hammered out the following short Perl script to download the patches:

use strict;
use warnings;
use WWW::Mechanize;
use WWW::Mechanize::TreeBuilder;

$|++;

my $base_url = "https://patchstorage.com/platform/vcv-rack/page/";
my $mech = WWW::Mechanize->new(autocheck=>0);
WWW::Mechanize::TreeBuilder->meta->apply($mech);
use constant SLEEP_TIME => 2;

my $seq = 1;
my $working = 1;
while ($working) {
  print "page $seq\n";
  $mech->get($base_url.$seq);
  sleep(SLEEP_TIME);
  my @patch_pages = $mech->look_down('_tag', 'a');
  my @patch_links = grep {
    defined $_ and
    !m[/upload\-a\-patch\/] and
    !m[/login/] and
    !m[/new\-tutorial/] and
    !m[/explore/] and
    !m[/registration/] and
    !m[/new\-question/] and
    !m[/explore/] and
    !m[/platform/] and
    !m[/tag/] and
    !m[/author/] and
    !m[/wp\-content/] and
    !m[/category/] and
    !/\#$/ and
    !/\#respond/ and
    !/\#comments/ and
    !/mailto:/ and
    !/\/privacy\-policy/ and
    !/discord/ and
    !/https:\/\/vcvrack/ and
    !/javascript:/ and
    !/action=lostpassword/ and
    !/patchstorage.com\/$/ and
    ! $_ eq ''} map {$_->attr('href')} @patch_pages;
    my %links;
    @links{@patch_links} = ();
    @patch_links = keys %links;
    print scalar @patch_links, " links found\n";
    for my $link (@patch_links) {
      next unless $link;
      print $link;
      my @parts = split /\//, $link;
      my $patch_name = $parts[-1];
      if (-f "/Users/jmcmahon/Downloads/$patch_name") {
        print "...skipped\n";
        next;
      }
      print "\n";
      $mech->get($link);
      sleep(SLEEP_TIME);
      my @patches = $mech->look_down('id', "DownloadPatch");
      for my $patch (@patches) {
        my $p_link = $patch->attr('href');
        next unless $p_link;
        print "$patch_name...";
        $mech->get($patch->attr('href'));
        sleep(SLEEP_TIME);
        open my $fh, ">", "/Users/jmcmahon/Downloads/$patch_name" or die "Can't open $patch_name: $!";
        print $fh $mech->content;
        close $fh;
        print "saved\n";
      }
    }
    $seq++;
 }

Notable items here:

• The infinite scroll is actually a chunk of Javascript wrapped around a standard WordPress page setup, so I can “page” back through the patches for Rack by incrementing the page number and pulling off the links to the actual posts with the patches in them.
• That giant grep and map cleans up the links I get off the individual pages to just the ones that are actually links to patches.
• I have a couple checks in there for “have I already downloaded this?” to allow me to restart the script if it dies partway through the process.
• The script kills itself off once it gets a page with no links on it. I haven’t actually gotten that far yet, but I think it should work.

Patchstorage folks: I apologize for scraping the site, but this is for my own use only; I”m not republishing. If I weren’t desperate to retrieve the patch for Friday I would have just left it alone.

I was clearing out my CPAN RT queue today, and found a question in the tickets for Test::Mock::LWP from dcantrell:

It’s not at all clear how to use this module. I have a module which (partly) wraps around LWP::UserAgent which I use to fetch data which my module then manipulates. Obviously I need to test that my module handles webby errors correctly, for instance that it correctly detects when the remote sites don’t respond; and I need to be able to feed known data to my module so I can test that it does those manipulations correctly.

Test::Mock::LWP is the obvious candidate for faking up LWP::UserAgent, but I just can’t figure out how to use it. Please can you write a HOWTO and add it to the docs.

I’m adding the HOWTO tonight, even though the question was asked 12 years ago (I really need to get to my RT queue more often). The module’s description as it stands is pretty opaque; this explanation should, I hope, make it much more clear.

HOWTO use Test::Mock::LWP

Test::Mock::LWP is designed to provide you a quick way to mock out LWP calls.

Exported variables

Test::Mock::LWP‘s interface is exposed via the variables it exports:

• $Mock_ua – mocks LWP::USerAgent
• $Mock_req / $Mock_request – mocks HTTP::Request
• $Mock_resp / $Mock_response – mocks HTTP::Response

All of these are actually Test::MockObject objects, so you call mock() on them to change how they operate dynamically. Here’s an example.

Let’s say you wanted the next response to an LWP call to return the content foo and an HTTP status code of 201. You’d do this:

 
BEGIN {
  # Load the mock modules *first*.
  use Test::Mock::LWP::UserAgent;
  use Test::Mock::HTTP::Response;
  use Test::Mock::HTTP::Request;
}

# Load the modules you'll use to actually do LWP operations.
# These will automatically be mocked for you.
use LWP::UserAgent;
use HTTP::Response;
use HTTP::Request;

# Now set up the response you want to get back.
$Mock_resp->mock( content => sub { 'foo' });
$Mock_resp->mock( code    => sub { 201 });

# Pretend we're making a request to a site.
for (1..2) {
  my $req   = HTTP::Request->new(GET => 'http://nevergoeshere.com');
  my $agent = LWP::UserAgent->new;
  my $res   = $agent->simple_request($req);
}
# The values you added to the mock are now there.
printf("The site returned %d %s\n", $res->code, $res->content);

This will print

201 foo
201 foo

Getting more than one value out of the mocks: repeated re-mocks

Note that the values are constrained to what you’ve sent to the mocks. The mock here will simply keep returning 201 and foo for as many times as you call it. You’ll need to re-mock the content and code methods
each time you want to change them.

my $req   = HTTP::Request->new(GET => 'http://nevergoeshere.com');
my $agent = LWP::UserAgent->new;

$Mock_resp->mock( content => sub { 'foo' });
$Mock_resp->mock( code    => sub { 201 });
my $res   = $agent->simple_request($req);

printf("The site returned %d %s\n", $res->code, $res->content);
# 201 foo
		
$Mock_resp->mock( content => sub { 'bar' });
$Mock_resp->mock( code    => sub { 400 });
my $res   = $agent->simple_request($req);

printf("The site returned %d %s\n", $res->code, $res->content);
# 400 bar	

Moving the logic into the mocks

If you have a fixed sequence of items to return, just add them all to the mocks and have the mocks step through them. Here’s an example where we hand off two lists of values to the mocks:

use strict;
BEGIN {
  # Load the mock modules *first*.
  use Test::Mock::LWP::UserAgent;
  use Test::Mock::HTTP::Response;
  use Test::Mock::HTTP::Request;
}

# Load the modules you'll use to actually do LWP operations.
# These will automatically be mocked for you.
use LWP::UserAgent;
use HTTP::Response;
use HTTP::Request;

my @contents = qw(foo bar baz);
my @codes    = qw(404 400 200);

# initialize counter.
my $code_counter = 2;
my $content_counter = 2;

my $content_sub = sub {
  $content_counter += 1;
  $content_counter %= 3;
  $contents[$content_counter];
};

my $code_sub = sub {
  $code_counter += 1;
  $code_counter %= 3;
  return $codes[$code_counter];
};
    
$Mock_resp->mock(content => $content_sub);
$Mock_resp->mock(code    => $code_sub);
    
my $req   = HTTP::Request->new(GET => 'http://nevergoeshere.com');
my $agent = LWP::UserAgent->new;
    
for (0..5) {
  my $res   = $agent->simple_request($req);
  printf("The site returned %d %s\n", $res->code, $res->content);
}

This will print

    The site returned 404 foo
    The site returned 400 bar
    The site returned 200 baz
    The site returned 404 foo
    The site returned 400 bar
    The site returned 200 baz

Remember: the key is make sure that the mock is ready to return the next item when you make the next request to the user agent.