To Be GNU/Hurd

Mon, 09 Jun 2008

hellasmella 0.11 fixes broken newzbin searching

Newzbin recently switched to v3 for the default interface, thus breaking the screen scraping that was being done. Here is a new version that forces the v2 interface.

HellaSmella-0.11.tar.gz

Comments

Jon
I am unable to add nzbs via newzbin id. Normal upload of nzb file works fine. When attempting to add via the newzbin id I can see the following output, but it doesn't add the nzb, any ideas?

[info] *** Request 262 (0.151/s) [23849] [Sat Jun 21 10:47:07 2008] ***
[debug] Body Parameters are:
.------------------------------------- --------------------------------------.
| Parameter  | Value  |
------------------------------------- --------------------------------------
| msgid  | 3066195  |
'------------------------------------- --------------------------------------'
[debug] "POST" request for "add" from "192.168.0.101"
[debug] Path is "add"
[debug] Checking http basic authentication.
[debug] Successfully authenticated user 'Catalyst::Authentication::User::Hash=HASH(0x8e785e4)'.
[info] Request took 0.089116s (11.221/s)
.---------------------------------------------------------------- -----------.
| Action  | Time  |
---------------------------------------------------------------- -----------
| /auto  | 0.002472s |
| /add  | 0.065029s |
|  -> /status  | 0.032490s |
|  -> HellaSmella::View::JSON->process  | 0.004549s |
| /end  | 0.001374s |
'---------------------------------------------------------------- -----------'
Lee
Before I investigate further, have you filled in the correct Newzbin login info into your hellasmella.yml file? You'll also need to make sure that you have Newzbin "credits" associated with your account.
Lee
You could also check if you can add them from the command line with this command:

hellanzb.py enqueuenewzbin <nzbid>
Kevin
Hi there,

I have been trying to install your sweet looking frontend all day with little luck. I have run into a few issues. First is your configuration for FastCGI in lighttpd. I am already using FastCGI for php which is working good. But when I add what you have in your ex/ folder to my working config it makes all my php stuff stop working. I think it is cause by this portion ' "" => ' of your config as I guess with that it tries to use the hellasmella fastcgi backend for everything?

I decided to try and run hellasmella_test.pl just to see if it worked and if anything happened. This was the output:

Can't locate Catalyst/Plugin/StackTrace.pm in @INC (@INC contains: /usr/bin/perlbin/site/../lib /usr/lib/perl5/site_perl/5.10.0 /usr/share/perl5/site_perl/5.10.0 /usr/lib/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib/perl5/core_perl /usr/share/perl5/core_perl /usr/lib/perl5/current /usr/lib/perl5/site_perl/current .) at /usr/share/perl5/site_perl/5.10.0/Catalyst/Utils.pm line 270.
Compilation failed in require at /usr/share/perl5/site_perl/5.10.0/Catalyst/Test.pm line 90.

As far as I know all the required modules installed when I was making hellasmella. Lastly I wasn't quite sure where to put the /root folder you mentioned in the /ex config. currently I have it like this:

/home/lighhtpd/html/HellaSmella/root

is that correct? It is where I put all my other webapps and they are working will. Would I think point my browser to http://myip/HellaSmella for this to work? Sprry for all the questions but I seem to be lost here. This is my first foray in Perl for web stuff and it's not going well.

Thanks,

Kevin
Jeanb
Hi,

I tried to install HellaSmella using Apache and FastCGI on a virtualhost but whatever I type in login and password (even the default hellanzb/hell) the auth request loops.

FastCGI: server "/usr/local/bin/hellasmella_fastcgi.pl" stderr: [info] *** Request 2 (0.000/s) [18672] [Mon Jul 28 00:30:38 2008] ***
FastCGI: server "/usr/local/bin/hellasmella_fastcgi.pl" stderr: [debug] "GET" request for "/" from "90.17.159.152"
FastCGI: server "/usr/local/bin/hellasmella_fastcgi.pl" stderr: [debug] Checking http basic authentication.
FastCGI: server "/usr/local/bin/hellasmella_fastcgi.pl" stderr: [info] Request took 0.309255s (3.234/s)
FastCGI: server "/usr/local/bin/hellasmella_fastcgi.pl" stderr: .---------------------------------------------------------------- -----------.
FastCGI: server "/usr/local/bin/hellasmella_fastcgi.pl" stderr: | Action  | Time  |
FastCGI: server "/usr/local/bin/hellasmella_fastcgi.pl" stderr:  ---------------------------------------------------------------- -----------
FastCGI: server "/usr/local/bin/hellasmella_fastcgi.pl" stderr: | /auto  | 0.003447s |
FastCGI: server "/usr/local/bin/hellasmella_fastcgi.pl" stderr: | /index  | 0.277450s |
FastCGI: server "/usr/local/bin/hellasmella_fastcgi.pl" stderr: |  -> /status  | 0.275078s |
FastCGI: server "/usr/local/bin/hellasmella_fastcgi.pl" stderr: | /end  | 0.002702s |
FastCGI: server "/usr/local/bin/hellasmella_fastcgi.pl" stderr: '---------------------------------------------------------------- -----------'


Don't know if I can verbose the log but It's all I've got :)
Jeanb
Ok I made it worked !

The problem is that FastCGI module doesn't transfert authorization header so the HTTP Authentication module is not working.

To solve that add (after have loaded the rewrite module) :
RewriteEngine On
RewriteCond %{HTTP:Authorization} ^(. )
RewriteRule ^(.*)$ $1 [E=HTTP_AUTHORIZATION:%1,PT]
Jeanb
Here is my config file :

<VirtualHost *:80>
  ServerAdmin webmaster@localhost

  ServerName ng.yourdomain.com
  ServerAlias ng

  DocumentRoot /usr/local/share/perl/5.10.0/HellaSmella/root/

  Alias /static /usr/local/share/perl/5.10.0/HellaSmella/root/static

  <Location /static>
  SetHandler default-handler
  </Location>


  FastCgiServer /usr/local/bin/hellasmella_fastcgi.pl -processes 3
  Alias / /usr/local/bin/hellasmella_fastcgi.pl/

  ErrorLog /var/log/apache2/error.log

  # Possible values include: debug, info, notice, warn, error, crit,
  # alert, emerg.
  LogLevel warn

  CustomLog /var/log/apache2/access.log combined

  RewriteEngine On
  RewriteCond %{HTTP:Authorization} ^(. )
  RewriteRule ^(.*)$ $1 [E=HTTP_AUTHORIZATION:%1,PT]
</VirtualHost>
Lee Aylward
Interesting, I hadn't done any FastCGI testing on Apache. I'll make HTTP Auth optional in the next release and work on a better solution.

Thanks for posting your solution!
gringo
I had a lot of trouble, but it seems nearly working.

I got a Window in my Browser to enter User and Pass.

Hard to get it working, because i am not fit in Perl.

After i did this i got this Error:



Caught exception in HellaSmella::Controller::Root->auto "Can't use string ("ingo") as a HASH ref while "strict refs" in use at /usr/local/share/perl/5.8.8/Catalyst/Authentication/Store/Minimal.pm line 38."

Stack Trace
Package Line File
HellaSmella::Controller::Root 33 /usr/local/share/perl/5.8.8/HellaSmella/Controller/Root.pm

30: 31: sub auto : Private { 32: my ( $self, $c ) = @_; 33: if ( ! $c->authenticate_http ) { 34: $c->authorization_required_response( realm => "HellaSmella" ); 35: } 36: $c->stash->{categories} = $c->model('Newzbin')->categories;

Request

bless({
  action  => "index",
  address  => "192.168.6.100",
  arguments  => [],
  base  => bless(do{\(my $o = "http://192.168.6.105/")}, "URI::http"),
  body_parameters  => {},
  captures  => [],
  cookies  => {
  1187  => bless({ name => 1187, path => "/", value => [1] }, "CGI::Simple::Cookie"),
  1191  => bless({ name => 1191, path => "/", value => [1] }, "CGI::Simple::Cookie"),
  1195  => bless({ name => 1195, path => "/", value => [1] }, "CGI::Simple::Cookie"),
  1199  => bless({ name => 1199, path => "/", value => [1] }, "CGI::Simple::Cookie"),
  1231  => bless({ name => 1231, path => "/", value => [1] }, "CGI::Simple::Cookie"),
  1237  => bless({ name => 1237, path => "/", value => [1] }, "CGI::Simple::Cookie"),
  1243  => bless({ name => 1243, path => "/", value => [1] }, "CGI::Simple::Cookie"),
  1246  => bless({ name => 1246, path => "/", value => [1] }, "CGI::Simple::Cookie"),
  3  => bless({ name => 3, path => "/", value => [0] }, "CGI::Simple::Cookie"),
  "tntnet." => bless({
  name  => "tntnet.",
  path  => "/",
  value => ["681724f0eba4a8485bcd3d1b9beaedb4"],
  }, "CGI::Simple::Cookie"),
  },
  headers  => bless({
  "accept" => "text/html,application/xhtml xml,application/xml;q=0.9,*/*;q=0.8",
  "accept-charset" => "ISO-8859-1,utf-8;q=0.7,*;q=0.7",
  "accept-encoding" => "gzip,deflate",
  "accept-language" => "de-de,de;q=0.8,en-us;q=0.5,en;q=0.3",
  authorization => "Basic ImluZ28iOmxpbmVhcg==",
  connection => "keep-alive",
  cookie => "1195=1; 1199=1; 3=0; 1187=1; 1191=1; 1246=1; 1243=1; 1237=1; 1231=1; tntnet.=681724f0eba4a8485bcd3d1b9beaedb4",
  host => "192.168.6.105",
  "keep-alive" => 300,
  "user-agent" => "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.0.1) Gecko/2008070208 Firefox/3.0.1",
  }, "HTTP::Headers"),
  hostname  => undef,
  match  => "",
  method  => "GET",
  parameters  => {},
  path  => "",
  protocol  => "HTTP/1.1",
  query_parameters => {},
  secure  => 0,
  uploads  => {},
  uri  => bless(do{\(my $o = "http://192.168.6.105/")}, "URI::http"),
  user  => undef,
}, "Catalyst::Request")

Response

bless({
  body  => "",
  cookies => {},
  headers => bless({
  "content-type" => "text/html; charset=utf-8",
  "x-catalyst"  => "5.7014",
  }, "HTTP::Headers"),
  status  => 200,
}, "Catalyst::Response")

Stash

{}

Config

do {
  my $a = {
  "Model::Hellanzb" => { location => "http://hellanzb:hella\@127.0.0.1:8760" },
  "Model::Newzbin" => { password => "pass", username => "user" },
  "Plugin::Authentication" => {
  http => { type => "basic" },
  password => "linear",
  stores => {},
  use_session => 1,
  users => "ingo",
  },
  "Plugin::ConfigLoader" => {},
  authentication => 'fix',
  debug => {
  ignore_classes => [
  "DBIx::Class::ResultSource::Table",
  "DBIx::Class::ResultSourceHandle",
  "DateTime",
  ],
  scrubber_func  => sub { "???" },
  },
  default_view => "TT",
  home => "/usr/local/share/perl/5.8.8/HellaSmella",
  name => "HellaSmella",
  nzb_directory => "/media/hellahella/nzb/daemon.queue/",
  root => bless({
  dirs => ["", "usr", "local", "share", "perl", "5.8.8", "HellaSmella", "root"],
  file_spec_class => undef,
  volume => "",
  }, "Path::Class::Dir"),
  stacktrace => { context => 3, verbose => 0 },
  static => {
  debug => 1,
  dirs => [],
  ignore_dirs => [],
  ignore_extensions => ["tmpl", "tt", "tt2", "html", "xhtml"],
  include_path => ['fix'],
  mime_types => {},
  mime_types_obj => bless({}, "MIME::Types"),
  no_logs => 1,
  },
  };
  $a->{authentication} = $a->{"Plugin::Authentication"};
  $a->{static}{include_path}[0] = $a->{root};
  $a;
}

HellaSmella on Catalyst 5.7014
pick420
Is there any way that you can update this to crawl the v3 newzbin site? For some reason I cannot do any searches what so ever with hellesmella when I try the newzbin search. I know I have the right password and user in the yml file. I even hard coded it to make sure at one point. I can probably do it myself but I'm a bit rusty on my perl at the moment. Please let me know, thanks
infty
Hi.

Is it possible to do bookmarklet queueing, just like hellahella? That's a good feature that I really miss from hellahella (which has gone tits-up for me using python 2.5.)

It'd be great to be able to queue nzbs from within newzbin.com by just clicking a bookmarklet...

cheers.
Lee
infty,

That should be very simple since hellasmella has a URL that accepts a Newzbin NZB ID to queue items. Any chance you could email me the hellahella bookmarklet?
Lee
pick420,

I had some trouble porting New::Newzbin to v3 a while ago due to some cookie issues. I'll take another look in the coming weeks/months.
lee
gringo,

Could you email me your hellasmella.yml file? From the error message it looks like there may be an error in there.
infty
lee,

I've emailed you the bookmarklet. Let me know if you need anything else.

Best,
infty
lee - can you let me know here or via email what the url is that hellasmella uses to accept nzbids? that way I can at least use it with some functionality already.

cheers.
Lee
infty,

The address is [% root %]/add?msgid=[% msgid %]

It does require you to be authenticated, so that may prove to be an issue. Maybe you can include the credentials in the URL string (e.g. http://user:pass@localhost/add?msgid=123456).

Good luck
infty
lee -

as a bookmarklet, this seems to work:

javascript:location.href='http://example.com/add?msgid=' encodeURIComponent(location.href.match(/[0-9] /))

Clearly it assumes that you are on a site whose address is of format
http://www.newzbin.com/browse/post/123456/

Caveat: match() returns an array. I'm not experienced at javascript, so I don't know if it just flattens the array here, or if this will go tits-up when there's more than one set of numbers in the title (i.e. http://v3.newzbin.com/browse/post/123456/). As far as I can tell, in this case, it just doesn't do anything.

The obvious improvement would be to add a new address thing, "nzbadd" or whatever, which can successfully parse the full address to extract the report ID.

Another possible improvement is to send the user back to the newzbin site after queueing with an automated redirect or something, but that's just aesthetics.

Hope this helps.
infty
oops - looks like something broke. there's a plus sign after the square brackets in the regular expression of course.
cb
Just wanted to let you know that installation is smooth and painless with Ubuntu.

aptitude install libcatalyst-engine-apache-perl libcatalyst-modules-extra-perl libcatalyst-modules-perl libcatalyst-perl libcatalyst-view-tt-perl libjson-any-perl par2 unrar libstring-escape-perl

Then HellaSmella will want to download, build, and install only these two packages which don't seem to be in any standard repositories:

- Catalyst::Model::XMLRPC 
- Catalyst::Plugin::Authentication::Credential::HTTP

Submit a new comment




E-mail:


URL:


Comment: