egg-tcls: fetch_weather.pl annotate

annotate fetch_weather.pl @ 158:db2b1f74d994

Initial import of weather data fetching backend, written in Perl.

author	Matti Hamalainen <ccr@tnsp.org>
date	Mon, 02 Jun 2014 12:29:11 +0300
parents
children	4f8a163b2bc1

rev	line source
158 db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	1 #!/usr/bin/perl -w
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	2 use strict;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	3 use LWP::UserAgent;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	4 use HTML::Entities;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	5 use XML::Simple;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	6 use Text::Iconv;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	7 use Date::Parse;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	8 use Data::Dumper;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	9
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	10 sub str_trim($)
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	11 {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	12 my $str = $_[0];
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	13 if (defined($str)) {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	14 $str =~ s/^\s*//;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	15 $str =~ s/\s*$//;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	16 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	17 return $str;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	18 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	19
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	20
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	21 sub pop_token_a($)
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	22 {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	23 my $tokens = $_[0];
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	24 return shift(@$tokens);
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	25 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	26
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	27
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	28 sub pop_token($)
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	29 {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	30 return str_trim(pop_token_a($_[0]));
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	31 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	32
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	33
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	34 sub parse_html_str($)
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	35 {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	36 my $tokens = $_[0];
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	37 my $token = pop_token($tokens);
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	38 my $str = "";
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	39 $token =~ s/^\s*//;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	40
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	41 return undef unless (substr($token, 0, 1) eq '"');
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	42 $token = substr($token, 1);
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	43
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	44 while (defined($token)) {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	45 my $tmp = $token;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	46 $tmp =~ s/\s*$//;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	47 if (substr($tmp, -1) eq '"') {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	48 $str .= substr($tmp, 0, -1);
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	49 return $str;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	50 } else {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	51 $str .= $token;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	52 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	53 $token = shift(@$tokens);
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	54 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	55 return undef;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	56 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	57
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	58
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	59 sub parse_html_tree($$);
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	60
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	61 sub parse_html_tree($$)
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	62 {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	63 my ($tokens, $tree) = @_;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	64
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	65 while (my $token = pop_token($tokens)) {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	66 if ($token =~ /^<[!\/]?[a-zA-Z]+/) {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	67 $token = lc($token);
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	68 if ($token =~ /^<\!.*>$/) {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	69 # Ignore comments etc.
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	70 } elsif ($token =~ /^<([a-z]+[1-9])(.)>$/) {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	71 my ($name, $args) = ($1, $2);
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	72 if ($name eq "meta" \|\| $name eq "img") {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	73 my $tmp = {};
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	74 parse_html_tree($tokens, $tree);
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	75 return $tree;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	76 } else {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	77 my $tmp = { "name" => $name, "args" => str_trim($args) };
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	78 parse_html_tree($tokens, $tmp);
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	79 push(@{$$tree{"nodes"}}, $tmp);
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	80 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	81 } elsif ($token =~ /^<\/([a-z]+[1-9]*)>$/) {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	82 return $tree;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	83 } else {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	84 die("HORROR TERROR ELITE: '$token'\n");
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	85 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	86 } else {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	87 $token = str_trim(decode_entities($token));
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	88 push(@{$$tree{"nodes"}}, { "name" => "text", "args" => "", "text" => $token }) if length($token) > 0;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	89 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	90 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	91
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	92 return $tree;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	93 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	94
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	95
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	96 sub parse_html($)
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	97 {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	98 return undef unless defined($_[0]);
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	99 my $str = $_[0];
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	100 my $res = { "name" => "", "args" => "" };
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	101 $str =~ tr/\r/ /;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	102 $str =~ tr/\n/ /;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	103 my @tokens = grep { !($_ =~ /^\s$/) } split(/(<\/?[a-zA-Z]+.?>)/, $str);
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	104 parse_html_tree(\@tokens, $res);
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	105 return $res;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	106 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	107
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	108 sub html_find_node($$$);
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	109
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	110 sub html_find_node($$$)
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	111 {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	112 my ($node, $name, $args) = @_;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	113
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	114 if (defined($node)) {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	115 if (ref($node) eq "ARRAY") {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	116 foreach my $n (@$node) {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	117 my $tmp = html_find_node($n, $name, $args);
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	118 # Must do it like this, in order not to break the loop
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	119 return $tmp if defined($tmp);
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	120 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	121 } elsif (ref($node) eq "HASH") {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	122 if (defined($$node{"name"})) {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	123 if ($$node{"name"} eq $name) {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	124 if ($args ne "") {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	125 if (defined($$node{"args"}) && $$node{"args"} =~ /$args/) {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	126 } else {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	127 return html_find_node($$node{"nodes"}, $name, $args);
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	128 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	129 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	130 return $node;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	131 } else {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	132 return html_find_node($$node{"nodes"}, $name, $args);
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	133 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	134 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	135 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	136 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	137 return undef;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	138 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	139
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	140
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	141 sub fnode($$)
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	142 {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	143 return html_find_node($_[0], $_[1], "");
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	144 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	145
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	146
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	147 sub fnodea($$$)
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	148 {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	149 return html_find_node($_[0], $_[1], $_[2]);
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	150 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	151
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	152
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	153 sub fetch_http($)
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	154 {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	155 my $tmpAgent = LWP::UserAgent->new;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	156 $tmpAgent->agent("Mozilla/4.0 (compatible; MSIE 6.0; MSIE 5.5; Windows NT 6.0) Opera 10.63 [en]");
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	157 $tmpAgent->timeout(10);
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	158
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	159 my $tmpRequest = HTTP::Request->new(GET => $_[0]);
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	160
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	161 return $tmpAgent->request($tmpRequest);
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	162 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	163
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	164
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	165 sub get_node($$$)
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	166 {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	167 return defined($_[0]->[$_[2]]{"nodes"}[0]{$_[1]}) ? $_[0]->[$_[2]]{"nodes"}[0]{$_[1]} : "";
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	168 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	169
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	170
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	171 sub parse_timestamp($$)
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	172 {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	173 return $_[1] + str2time($_[0]) - time();
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	174 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	175
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	176
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	177 ###
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	178 ### Main program begins
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	179 ###
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	180 die(
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	181 "Weather Fetch v0.1 by ccr/TNSP <ccr\@tnsp.org>\n".
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	182 "Usage: $0 <output_data_file> <fmi_api_key>\n"
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	183 ) unless scalar(@ARGV) >= 2;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	184
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	185 my $opt_outfile = shift;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	186 my $opt_api_key = shift;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	187 my $weatherdata = {};
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	188
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	189 ### Fetch tiehallinto data
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	190 for (my $i = 1; $i <= 22; $i++)
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	191 {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	192 my $res = fetch_http("http://alk.tiehallinto.fi/alk/tiesaa/tiesaa_maak_".$i.".html");
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	193 if ($res->code >= 200 && $res->code <= 201)
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	194 {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	195 my $data = $res->decoded_content;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	196
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	197 # Filter out crap tags we don't want or need
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	198 $data =~ s/\n/§/g;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	199 $data =~ s/<!--.*?-->//ig;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	200 $data =~ s/<map[^>]>.?<\/map>//ig;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	201 $data =~ s/<form[^>]>.?<\/form>//ig;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	202 $data =~ s/<script[^>]>.?<\/script>//ig;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	203 $data =~ s/<meta[^>]*>//ig;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	204 $data =~ s/<font[^>]*>//ig;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	205 $data =~ s/<\/font>//ig;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	206 $data =~ s/<span[^>]*>//ig;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	207 $data =~ s/<\/span>//ig;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	208 $data =~ s/<\/?b>//ig;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	209
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	210 $data =~ s/<br>//ig;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	211 $data =~ s/ / /ig;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	212 $data =~ s/§/\n/g;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	213
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	214 # Parse the HTML mess
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	215 my $otree = parse_html($data);
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	216
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	217 # Find our desired element nodes
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	218 my $odata = fnodea(fnode($otree, "body"), "div", "class=elementc");
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	219 my $oupdate = fnode($odata, "p");
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	220 my $time_base = time();
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	221 if ($oupdate) {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	222 my $tmp = $oupdate->{"nodes"}[0]{"text"};
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	223 if ($tmp =~ /:\s+(\d\d\.\d\d\.\d\d\d\d)\s+(\d\d:\d\d)/) {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	224 $time_base = str2time($1);
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	225 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	226 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	227
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	228 my $oelems = fnode($odata, "table")->{"nodes"};
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	229 for (my $n = 1; $n < scalar(@$oelems); $n++)
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	230 {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	231 my $fdata = @$oelems[$n]->{"nodes"};
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	232 $weatherdata->{$fdata->[0]{"nodes"}[0]{"text"}} =
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	233 [
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	234 parse_timestamp(get_node($fdata, "text", 1), $time_base),
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	235 get_node($fdata, "text", 2),
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	236 get_node($fdata, "text", 3),
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	237 get_node($fdata, "text", 4),
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	238 get_node($fdata, "text", 5),
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	239 ];
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	240 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	241 print STDERR ".";
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	242 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	243 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	244
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	245 ### Fetch FMI data
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	246 my $res = fetch_http("http://data.fmi.fi/fmi-apikey/".$opt_api_key.
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	247 "/wfs?request=getFeature&storedquery_id=fmi::observations::weather::cities::multipointcoverage".
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	248 "&parameters=temperature,humidity");
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	249
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	250 if ($res->code >= 200 && $res->code <= 201)
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	251 {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	252 my $xml = XMLin($res->decoded_content,
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	253 KeyAttr => { server => 'name' },
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	254 ForceArray => [ 'server', 'address' ]);
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	255 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	256
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	257 ### Output
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	258 if (defined($opt_outfile)) {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	259 open(STDOUT, '>', $opt_outfile) or die("Could not open output file '$opt_outfile'.\n");
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	260 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	261
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	262 binmode STDOUT, ':encoding(utf-8)';
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	263
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	264 foreach my $key (sort { $a cmp $b } keys %$weatherdata)
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	265 {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	266 print STDOUT $key."\|".join("\|", @{$weatherdata->{$key}})."\n";
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	267 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	268
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	269 close(STDOUT);

Mercurial > hg > egg-tcls

annotate fetch_weather.pl @ 158:db2b1f74d994