egg-tcls: fetch_weather.pl annotate

annotate fetch_weather.pl @ 213:330d7e39c8f4

weather: Add extra data field.

author	Matti Hamalainen <ccr@tnsp.org>
date	Wed, 24 Sep 2014 21:42:58 +0300
parents	5f04ec60e8d9
children	fe420576f284

rev	line source
158 db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	1 #!/usr/bin/perl -w
182 24f9f38210fe Add comments. Matti Hamalainen <ccr@tnsp.org> parents: 178 diff changeset	2 ##########################################################################
24f9f38210fe Add comments. Matti Hamalainen <ccr@tnsp.org> parents: 178 diff changeset	3 #
24f9f38210fe Add comments. Matti Hamalainen <ccr@tnsp.org> parents: 178 diff changeset	4 # Fetch Weather v0.1 by Matti 'ccr' Hamalainen <ccr@tnsp.org>
24f9f38210fe Add comments. Matti Hamalainen <ccr@tnsp.org> parents: 178 diff changeset	5 # (C) Copyright 2014 Tecnic Software productions (TNSP)
206 2ac661d551b3 Update documentation. Matti Hamalainen <ccr@tnsp.org> parents: 201 diff changeset	6 # This script is freely distributable under GNU GPL (version 2) license.
182 24f9f38210fe Add comments. Matti Hamalainen <ccr@tnsp.org> parents: 178 diff changeset	7 #
24f9f38210fe Add comments. Matti Hamalainen <ccr@tnsp.org> parents: 178 diff changeset	8 # Should be ran as a cronjob, and configured properly.
206 2ac661d551b3 Update documentation. Matti Hamalainen <ccr@tnsp.org> parents: 201 diff changeset	9 # /10 * * * perl -w /absolute/path/to/fetch_weather.pl /path/to/configfile
182 24f9f38210fe Add comments. Matti Hamalainen <ccr@tnsp.org> parents: 178 diff changeset	10 #
186 b41c525bdaf7 Add example configuration file. Matti Hamalainen <ccr@tnsp.org> parents: 185 diff changeset	11 # Configuration file example is in fetch_weather.config
182 24f9f38210fe Add comments. Matti Hamalainen <ccr@tnsp.org> parents: 178 diff changeset	12 #
206 2ac661d551b3 Update documentation. Matti Hamalainen <ccr@tnsp.org> parents: 201 diff changeset	13 # Requires various Perl modules, in Debian the packages should be:
2ac661d551b3 Update documentation. Matti Hamalainen <ccr@tnsp.org> parents: 201 diff changeset	14 # libwww-perl libxml-simple-perl libtimedate-perl
2ac661d551b3 Update documentation. Matti Hamalainen <ccr@tnsp.org> parents: 201 diff changeset	15 #
182 24f9f38210fe Add comments. Matti Hamalainen <ccr@tnsp.org> parents: 178 diff changeset	16 #
24f9f38210fe Add comments. Matti Hamalainen <ccr@tnsp.org> parents: 178 diff changeset	17 ##########################################################################
158 db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	18 use strict;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	19 use LWP::UserAgent;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	20 use HTML::Entities;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	21 use XML::Simple;
189 3af8ea19a3e2 Improved weather fetching. Matti Hamalainen <ccr@tnsp.org> parents: 186 diff changeset	22 use Date::Format;
199 95012f8f687a Another round of timestamp parsing changes, urgh. Matti Hamalainen <ccr@tnsp.org> parents: 198 diff changeset	23 use Date::Parse;
158 db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	24 use Data::Dumper;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	25
166 c8d5f3a7c4b7 Add debug option. Matti Hamalainen <ccr@tnsp.org> parents: 164 diff changeset	26
162 4f8a163b2bc1 Work on the backend weather data fetching. Matti Hamalainen <ccr@tnsp.org> parents: 158 diff changeset	27 ###
182 24f9f38210fe Add comments. Matti Hamalainen <ccr@tnsp.org> parents: 178 diff changeset	28 ### Configuration settings
162 4f8a163b2bc1 Work on the backend weather data fetching. Matti Hamalainen <ccr@tnsp.org> parents: 158 diff changeset	29 ###
164 d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	30 my %settings = (
166 c8d5f3a7c4b7 Add debug option. Matti Hamalainen <ccr@tnsp.org> parents: 164 diff changeset	31 "debug" => 0,
164 d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	32 "opt_fmi" => 0,
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	33 "opt_tiehallinto" => 0,
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	34 "fmi_api_key" => "",
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	35 "outfile" => "",
185 64e06ee172c7 Add HTTP user agent setting. Matti Hamalainen <ccr@tnsp.org> parents: 182 diff changeset	36 "http_user_agent" => "Mozilla/4.0 (compatible; MSIE 6.0; MSIE 5.5; Windows NT 6.0) Opera 10.63 [en]",
164 d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	37 );
162 4f8a163b2bc1 Work on the backend weather data fetching. Matti Hamalainen <ccr@tnsp.org> parents: 158 diff changeset	38
178 9b8ec700ede4 Clean up the weather data parser backend a bit. Matti Hamalainen <ccr@tnsp.org> parents: 170 diff changeset	39
162 4f8a163b2bc1 Work on the backend weather data fetching. Matti Hamalainen <ccr@tnsp.org> parents: 158 diff changeset	40 ###
4f8a163b2bc1 Work on the backend weather data fetching. Matti Hamalainen <ccr@tnsp.org> parents: 158 diff changeset	41 ### Helper functions
4f8a163b2bc1 Work on the backend weather data fetching. Matti Hamalainen <ccr@tnsp.org> parents: 158 diff changeset	42 ###
164 d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	43 sub mlog($)
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	44 {
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	45 print STDERR $_[0];
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	46 }
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	47
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	48
178 9b8ec700ede4 Clean up the weather data parser backend a bit. Matti Hamalainen <ccr@tnsp.org> parents: 170 diff changeset	49 sub fetch_http($)
9b8ec700ede4 Clean up the weather data parser backend a bit. Matti Hamalainen <ccr@tnsp.org> parents: 170 diff changeset	50 {
198 60a4c69c7f25 Cosmetics. Matti Hamalainen <ccr@tnsp.org> parents: 197 diff changeset	51 my $agent = LWP::UserAgent->new;
60a4c69c7f25 Cosmetics. Matti Hamalainen <ccr@tnsp.org> parents: 197 diff changeset	52 $agent->agent($settings{"http_user_agent"});
208 c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	53 $agent->timeout(20);
178 9b8ec700ede4 Clean up the weather data parser backend a bit. Matti Hamalainen <ccr@tnsp.org> parents: 170 diff changeset	54
198 60a4c69c7f25 Cosmetics. Matti Hamalainen <ccr@tnsp.org> parents: 197 diff changeset	55 my $req = HTTP::Request->new(GET => $_[0]);
178 9b8ec700ede4 Clean up the weather data parser backend a bit. Matti Hamalainen <ccr@tnsp.org> parents: 170 diff changeset	56
198 60a4c69c7f25 Cosmetics. Matti Hamalainen <ccr@tnsp.org> parents: 197 diff changeset	57 return $agent->request($req);
178 9b8ec700ede4 Clean up the weather data parser backend a bit. Matti Hamalainen <ccr@tnsp.org> parents: 170 diff changeset	58 }
9b8ec700ede4 Clean up the weather data parser backend a bit. Matti Hamalainen <ccr@tnsp.org> parents: 170 diff changeset	59
9b8ec700ede4 Clean up the weather data parser backend a bit. Matti Hamalainen <ccr@tnsp.org> parents: 170 diff changeset	60
9b8ec700ede4 Clean up the weather data parser backend a bit. Matti Hamalainen <ccr@tnsp.org> parents: 170 diff changeset	61 sub parse_timestamp($$)
9b8ec700ede4 Clean up the weather data parser backend a bit. Matti Hamalainen <ccr@tnsp.org> parents: 170 diff changeset	62 {
196 a38c123921a9 Change the timestamp parsing. Matti Hamalainen <ccr@tnsp.org> parents: 195 diff changeset	63 my ($str, $offs) = @_;
a38c123921a9 Change the timestamp parsing. Matti Hamalainen <ccr@tnsp.org> parents: 195 diff changeset	64 if ($str =~ /^(\d+):(\d+)$/)
a38c123921a9 Change the timestamp parsing. Matti Hamalainen <ccr@tnsp.org> parents: 195 diff changeset	65 {
a38c123921a9 Change the timestamp parsing. Matti Hamalainen <ccr@tnsp.org> parents: 195 diff changeset	66 return $offs + (60 * 60 * $1) + ($2 * 60);
a38c123921a9 Change the timestamp parsing. Matti Hamalainen <ccr@tnsp.org> parents: 195 diff changeset	67 }
a38c123921a9 Change the timestamp parsing. Matti Hamalainen <ccr@tnsp.org> parents: 195 diff changeset	68 else
a38c123921a9 Change the timestamp parsing. Matti Hamalainen <ccr@tnsp.org> parents: 195 diff changeset	69 {
a38c123921a9 Change the timestamp parsing. Matti Hamalainen <ccr@tnsp.org> parents: 195 diff changeset	70 return $offs;
a38c123921a9 Change the timestamp parsing. Matti Hamalainen <ccr@tnsp.org> parents: 195 diff changeset	71 }
178 9b8ec700ede4 Clean up the weather data parser backend a bit. Matti Hamalainen <ccr@tnsp.org> parents: 170 diff changeset	72 }
9b8ec700ede4 Clean up the weather data parser backend a bit. Matti Hamalainen <ccr@tnsp.org> parents: 170 diff changeset	73
9b8ec700ede4 Clean up the weather data parser backend a bit. Matti Hamalainen <ccr@tnsp.org> parents: 170 diff changeset	74
194 55ed3db51ee7 Rename a function. Matti Hamalainen <ccr@tnsp.org> parents: 193 diff changeset	75 sub format_time_gmt($)
189 3af8ea19a3e2 Improved weather fetching. Matti Hamalainen <ccr@tnsp.org> parents: 186 diff changeset	76 {
3af8ea19a3e2 Improved weather fetching. Matti Hamalainen <ccr@tnsp.org> parents: 186 diff changeset	77 # 2012-02-27T00:00:00Z
199 95012f8f687a Another round of timestamp parsing changes, urgh. Matti Hamalainen <ccr@tnsp.org> parents: 198 diff changeset	78 return time2str("%Y-%m-%dT%TZ", $_[0], "UTC");
189 3af8ea19a3e2 Improved weather fetching. Matti Hamalainen <ccr@tnsp.org> parents: 186 diff changeset	79 }
3af8ea19a3e2 Improved weather fetching. Matti Hamalainen <ccr@tnsp.org> parents: 186 diff changeset	80
193 93ec73deebc5 Cosmetics. Matti Hamalainen <ccr@tnsp.org> parents: 192 diff changeset	81
158 db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	82 sub str_trim($)
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	83 {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	84 my $str = $_[0];
193 93ec73deebc5 Cosmetics. Matti Hamalainen <ccr@tnsp.org> parents: 192 diff changeset	85 if (defined($str))
93ec73deebc5 Cosmetics. Matti Hamalainen <ccr@tnsp.org> parents: 192 diff changeset	86 {
158 db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	87 $str =~ s/^\s*//;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	88 $str =~ s/\s*$//;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	89 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	90 return $str;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	91 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	92
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	93
178 9b8ec700ede4 Clean up the weather data parser backend a bit. Matti Hamalainen <ccr@tnsp.org> parents: 170 diff changeset	94 ###
9b8ec700ede4 Clean up the weather data parser backend a bit. Matti Hamalainen <ccr@tnsp.org> parents: 170 diff changeset	95 ### Loose HTML parser
9b8ec700ede4 Clean up the weather data parser backend a bit. Matti Hamalainen <ccr@tnsp.org> parents: 170 diff changeset	96 ###
158 db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	97 sub pop_token_a($)
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	98 {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	99 my $tokens = $_[0];
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	100 return shift(@$tokens);
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	101 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	102
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	103
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	104 sub pop_token($)
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	105 {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	106 return str_trim(pop_token_a($_[0]));
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	107 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	108
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	109
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	110 sub parse_html_str($)
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	111 {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	112 my $tokens = $_[0];
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	113 my $token = pop_token($tokens);
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	114 my $str = "";
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	115 $token =~ s/^\s*//;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	116
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	117 return undef unless (substr($token, 0, 1) eq '"');
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	118 $token = substr($token, 1);
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	119
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	120 while (defined($token)) {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	121 my $tmp = $token;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	122 $tmp =~ s/\s*$//;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	123 if (substr($tmp, -1) eq '"') {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	124 $str .= substr($tmp, 0, -1);
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	125 return $str;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	126 } else {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	127 $str .= $token;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	128 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	129 $token = shift(@$tokens);
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	130 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	131 return undef;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	132 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	133
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	134
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	135 sub parse_html_tree($$);
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	136
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	137 sub parse_html_tree($$)
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	138 {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	139 my ($tokens, $tree) = @_;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	140
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	141 while (my $token = pop_token($tokens)) {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	142 if ($token =~ /^<[!\/]?[a-zA-Z]+/) {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	143 $token = lc($token);
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	144 if ($token =~ /^<\!.*>$/) {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	145 # Ignore comments etc.
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	146 } elsif ($token =~ /^<([a-z]+[1-9])(.)>$/) {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	147 my ($name, $args) = ($1, $2);
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	148 if ($name eq "meta" \|\| $name eq "img") {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	149 my $tmp = {};
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	150 parse_html_tree($tokens, $tree);
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	151 return $tree;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	152 } else {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	153 my $tmp = { "name" => $name, "args" => str_trim($args) };
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	154 parse_html_tree($tokens, $tmp);
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	155 push(@{$$tree{"nodes"}}, $tmp);
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	156 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	157 } elsif ($token =~ /^<\/([a-z]+[1-9]*)>$/) {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	158 return $tree;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	159 } else {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	160 die("HORROR TERROR ELITE: '$token'\n");
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	161 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	162 } else {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	163 $token = str_trim(decode_entities($token));
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	164 push(@{$$tree{"nodes"}}, { "name" => "text", "args" => "", "text" => $token }) if length($token) > 0;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	165 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	166 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	167
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	168 return $tree;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	169 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	170
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	171
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	172 sub parse_html($)
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	173 {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	174 return undef unless defined($_[0]);
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	175 my $str = $_[0];
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	176 my $res = { "name" => "", "args" => "" };
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	177 $str =~ tr/\r/ /;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	178 $str =~ tr/\n/ /;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	179 my @tokens = grep { !($_ =~ /^\s$/) } split(/(<\/?[a-zA-Z]+.?>)/, $str);
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	180 parse_html_tree(\@tokens, $res);
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	181 return $res;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	182 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	183
197 0fd2435371eb Cosmetics. Matti Hamalainen <ccr@tnsp.org> parents: 196 diff changeset	184
158 db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	185 sub html_find_node($$$);
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	186
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	187 sub html_find_node($$$)
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	188 {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	189 my ($node, $name, $args) = @_;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	190
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	191 if (defined($node)) {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	192 if (ref($node) eq "ARRAY") {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	193 foreach my $n (@$node) {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	194 my $tmp = html_find_node($n, $name, $args);
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	195 # Must do it like this, in order not to break the loop
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	196 return $tmp if defined($tmp);
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	197 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	198 } elsif (ref($node) eq "HASH") {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	199 if (defined($$node{"name"})) {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	200 if ($$node{"name"} eq $name) {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	201 if ($args ne "") {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	202 if (defined($$node{"args"}) && $$node{"args"} =~ /$args/) {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	203 } else {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	204 return html_find_node($$node{"nodes"}, $name, $args);
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	205 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	206 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	207 return $node;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	208 } else {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	209 return html_find_node($$node{"nodes"}, $name, $args);
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	210 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	211 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	212 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	213 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	214 return undef;
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	215 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	216
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	217
178 9b8ec700ede4 Clean up the weather data parser backend a bit. Matti Hamalainen <ccr@tnsp.org> parents: 170 diff changeset	218 ###
9b8ec700ede4 Clean up the weather data parser backend a bit. Matti Hamalainen <ccr@tnsp.org> parents: 170 diff changeset	219 ### Helper functions for locating/traversing nodes
9b8ec700ede4 Clean up the weather data parser backend a bit. Matti Hamalainen <ccr@tnsp.org> parents: 170 diff changeset	220 ### in the parsed data tree structure.
9b8ec700ede4 Clean up the weather data parser backend a bit. Matti Hamalainen <ccr@tnsp.org> parents: 170 diff changeset	221 ###
158 db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	222 sub fnode($$)
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	223 {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	224 return html_find_node($_[0], $_[1], "");
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	225 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	226
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	227
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	228 sub fnodea($$$)
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	229 {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	230 return html_find_node($_[0], $_[1], $_[2]);
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	231 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	232
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	233
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	234 sub get_node($$$)
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	235 {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	236 return defined($_[0]->[$_[2]]{"nodes"}[0]{$_[1]}) ? $_[0]->[$_[2]]{"nodes"}[0]{$_[1]} : "";
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	237 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	238
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	239
212 5f04ec60e8d9 weather: "Translate" rain state from Tiehallitus data. Matti Hamalainen <ccr@tnsp.org> parents: 208 diff changeset	240 my %th_rain_states =
5f04ec60e8d9 weather: "Translate" rain state from Tiehallitus data. Matti Hamalainen <ccr@tnsp.org> parents: 208 diff changeset	241 (
5f04ec60e8d9 weather: "Translate" rain state from Tiehallitus data. Matti Hamalainen <ccr@tnsp.org> parents: 208 diff changeset	242 "Pouta" => "Poutaa",
5f04ec60e8d9 weather: "Translate" rain state from Tiehallitus data. Matti Hamalainen <ccr@tnsp.org> parents: 208 diff changeset	243 "Heikko" => "Heikkoa sadetta",
5f04ec60e8d9 weather: "Translate" rain state from Tiehallitus data. Matti Hamalainen <ccr@tnsp.org> parents: 208 diff changeset	244 "Kohtalainen" => "Kohtalaista sadetta",
5f04ec60e8d9 weather: "Translate" rain state from Tiehallitus data. Matti Hamalainen <ccr@tnsp.org> parents: 208 diff changeset	245 "Voimakas" => "Voimakasta sadetta",
5f04ec60e8d9 weather: "Translate" rain state from Tiehallitus data. Matti Hamalainen <ccr@tnsp.org> parents: 208 diff changeset	246 );
5f04ec60e8d9 weather: "Translate" rain state from Tiehallitus data. Matti Hamalainen <ccr@tnsp.org> parents: 208 diff changeset	247
5f04ec60e8d9 weather: "Translate" rain state from Tiehallitus data. Matti Hamalainen <ccr@tnsp.org> parents: 208 diff changeset	248 my $th_rain_states_k = join("\|", map {quotemeta} sort { length($b)<=>length($a) } keys %th_rain_states);
5f04ec60e8d9 weather: "Translate" rain state from Tiehallitus data. Matti Hamalainen <ccr@tnsp.org> parents: 208 diff changeset	249
5f04ec60e8d9 weather: "Translate" rain state from Tiehallitus data. Matti Hamalainen <ccr@tnsp.org> parents: 208 diff changeset	250 sub translate_th_rain($)
5f04ec60e8d9 weather: "Translate" rain state from Tiehallitus data. Matti Hamalainen <ccr@tnsp.org> parents: 208 diff changeset	251 {
5f04ec60e8d9 weather: "Translate" rain state from Tiehallitus data. Matti Hamalainen <ccr@tnsp.org> parents: 208 diff changeset	252 my $tmp = $_[0];
5f04ec60e8d9 weather: "Translate" rain state from Tiehallitus data. Matti Hamalainen <ccr@tnsp.org> parents: 208 diff changeset	253 $tmp =~ s/($th_rain_states_k)/$th_rain_states{$1}/go;
5f04ec60e8d9 weather: "Translate" rain state from Tiehallitus data. Matti Hamalainen <ccr@tnsp.org> parents: 208 diff changeset	254 return $tmp;
5f04ec60e8d9 weather: "Translate" rain state from Tiehallitus data. Matti Hamalainen <ccr@tnsp.org> parents: 208 diff changeset	255 }
5f04ec60e8d9 weather: "Translate" rain state from Tiehallitus data. Matti Hamalainen <ccr@tnsp.org> parents: 208 diff changeset	256
5f04ec60e8d9 weather: "Translate" rain state from Tiehallitus data. Matti Hamalainen <ccr@tnsp.org> parents: 208 diff changeset	257
158 db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	258 ###
178 9b8ec700ede4 Clean up the weather data parser backend a bit. Matti Hamalainen <ccr@tnsp.org> parents: 170 diff changeset	259 ### Configuration handling
164 d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	260 ###
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	261 sub opt_chk_bool($)
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	262 {
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	263 if (defined($settings{$_[0]}))
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	264 {
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	265 my $val = $settings{$_[0]};
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	266 return ($val == 1 \|\| $val eq "true" \|\| $val eq "on" \|\| $val eq "1");
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	267 }
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	268 else
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	269 {
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	270 return 0;
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	271 }
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	272 }
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	273
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	274
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	275 sub opt_chk_valid($$)
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	276 {
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	277 if (defined($settings{$_[0]}))
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	278 {
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	279 my $val = $settings{$_[0]};
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	280 return length($val) >= $_[1];
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	281 }
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	282 else
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	283 {
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	284 return 0;
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	285 }
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	286 }
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	287
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	288
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	289 sub opt_get($)
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	290 {
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	291 if (defined($settings{$_[0]}))
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	292 {
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	293 return $settings{$_[0]};
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	294 }
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	295 else
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	296 {
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	297 return undef;
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	298 }
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	299 }
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	300
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	301
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	302 sub opt_read_config($)
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	303 {
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	304 my $filename = $_[0];
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	305 my $errors = 0;
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	306 my $line = 0;
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	307
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	308 open(CONFFILE, "<", $filename) or die("Could not open configuration '".$filename."'!\n");
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	309 while (<CONFFILE>)
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	310 {
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	311 $line++;
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	312 chomp;
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	313 if (/(^\s#\|^\s$)/) {
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	314 # Ignore comments and empty lines
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	315 } elsif (/^\s\"?([a-zA-Z0-9_]+)\"?\s=>?\s(\d+),?\s$/) {
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	316 my $key = lc($1);
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	317 my $value = $2;
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	318 if (defined($settings{$key})) {
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	319 $settings{$key} = $value;
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	320 } else {
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	321 mlog("[$filename:$line] Unknown setting '$key' = $value\n");
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	322 $errors = 1;
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	323 }
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	324 } elsif (/^\s\"?([a-zA-Z0-9_]+)\"?\s=>?\s\"(.?)\",?\s*$/) {
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	325 my $key = lc($1);
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	326 my $value = $2;
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	327 if (defined($settings{$key})) {
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	328 $settings{$key} = $value;
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	329 } else {
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	330 mlog("[$filename:$line] Unknown setting '$key' = '$value'\n");
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	331 $errors = 1;
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	332 }
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	333 } else {
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	334 mlog("[$filename:$line] Syntax error: $_\n");
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	335 $errors = 1;
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	336 }
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	337 }
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	338 close(CONFFILE);
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	339 return $errors;
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	340 }
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	341
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	342
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	343 ###
158 db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	344 ### Main program begins
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	345 ###
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	346 my $weatherdata = {};
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	347
164 d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	348 die(
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	349 "Weather Fetch v0.1 by ccr/TNSP <ccr\@tnsp.org>\n".
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	350 "Usage: $0 <config file>\n"
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	351 ) unless scalar(@ARGV) >= 1;
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	352
178 9b8ec700ede4 Clean up the weather data parser backend a bit. Matti Hamalainen <ccr@tnsp.org> parents: 170 diff changeset	353 my $cfgfile = shift;
9b8ec700ede4 Clean up the weather data parser backend a bit. Matti Hamalainen <ccr@tnsp.org> parents: 170 diff changeset	354 opt_read_config($cfgfile) == 0 or die("Errors while parsing configuration file '".$cfgfile."'.\n");
164 d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	355
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	356
178 9b8ec700ede4 Clean up the weather data parser backend a bit. Matti Hamalainen <ccr@tnsp.org> parents: 170 diff changeset	357 ###
9b8ec700ede4 Clean up the weather data parser backend a bit. Matti Hamalainen <ccr@tnsp.org> parents: 170 diff changeset	358 ### Fetch tiehallinto road weather measurement data
9b8ec700ede4 Clean up the weather data parser backend a bit. Matti Hamalainen <ccr@tnsp.org> parents: 170 diff changeset	359 ###
164 d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	360 if (opt_chk_bool("opt_tiehallinto"))
158 db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	361 {
164 d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	362 for (my $i = 1; $i <= 22; $i++)
158 db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	363 {
164 d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	364 my $res = fetch_http("http://alk.tiehallinto.fi/alk/tiesaa/tiesaa_maak_".$i.".html");
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	365 if ($res->code >= 200 && $res->code <= 201)
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	366 {
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	367 my $data = $res->decoded_content;
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	368
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	369 # Filter out crap tags we don't want or need
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	370 $data =~ s/\n/§/g;
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	371 $data =~ s/<!--.*?-->//ig;
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	372 $data =~ s/<map[^>]>.?<\/map>//ig;
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	373 $data =~ s/<form[^>]>.?<\/form>//ig;
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	374 $data =~ s/<script[^>]>.?<\/script>//ig;
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	375 $data =~ s/<meta[^>]*>//ig;
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	376 $data =~ s/<font[^>]*>//ig;
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	377 $data =~ s/<\/font>//ig;
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	378 $data =~ s/<span[^>]*>//ig;
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	379 $data =~ s/<\/span>//ig;
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	380 $data =~ s/<\/?b>//ig;
158 db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	381
164 d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	382 $data =~ s/<br>//ig;
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	383 $data =~ s/ / /ig;
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	384 $data =~ s/§/\n/g;
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	385
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	386 # Parse the HTML mess
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	387 my $otree = parse_html($data);
158 db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	388
164 d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	389 # Find our desired element nodes
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	390 my $odata = fnodea(fnode($otree, "body"), "div", "class=elementc");
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	391 my $oupdate = fnode($odata, "p");
199 95012f8f687a Another round of timestamp parsing changes, urgh. Matti Hamalainen <ccr@tnsp.org> parents: 198 diff changeset	392 my $time_base = str2time("00:00");
164 d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	393 if ($oupdate) {
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	394 my $tmp = $oupdate->{"nodes"}[0]{"text"};
199 95012f8f687a Another round of timestamp parsing changes, urgh. Matti Hamalainen <ccr@tnsp.org> parents: 198 diff changeset	395 if ($tmp =~ /:\s+(\d\d)\.(\d\d)\.(\d\d\d\d)\s+(\d\d:\d\d)/) {
95012f8f687a Another round of timestamp parsing changes, urgh. Matti Hamalainen <ccr@tnsp.org> parents: 198 diff changeset	396 $time_base = str2time($3."-".$2."-".$1);
164 d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	397 }
158 db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	398 }
164 d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	399
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	400 my $oelems = fnode($odata, "table")->{"nodes"};
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	401 for (my $n = 1; $n < scalar(@$oelems); $n++)
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	402 {
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	403 my $fdata = @$oelems[$n]->{"nodes"};
212 5f04ec60e8d9 weather: "Translate" rain state from Tiehallitus data. Matti Hamalainen <ccr@tnsp.org> parents: 208 diff changeset	404 $weatherdata->{get_node($fdata, "text", 0)} =
164 d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	405 [
201 92f4a489b7ef Add data type index. Matti Hamalainen <ccr@tnsp.org> parents: 200 diff changeset	406 0,
164 d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	407 parse_timestamp(get_node($fdata, "text", 1), $time_base),
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	408 get_node($fdata, "text", 2),
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	409 get_node($fdata, "text", 3),
212 5f04ec60e8d9 weather: "Translate" rain state from Tiehallitus data. Matti Hamalainen <ccr@tnsp.org> parents: 208 diff changeset	410 translate_th_rain(get_node($fdata, "text", 4)),
213 330d7e39c8f4 weather: Add extra data field. Matti Hamalainen <ccr@tnsp.org> parents: 212 diff changeset	411 "",
212 5f04ec60e8d9 weather: "Translate" rain state from Tiehallitus data. Matti Hamalainen <ccr@tnsp.org> parents: 208 diff changeset	412 get_node($fdata, "text", 5),
164 d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	413 ];
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	414 }
158 db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	415 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	416 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	417 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	418
178 9b8ec700ede4 Clean up the weather data parser backend a bit. Matti Hamalainen <ccr@tnsp.org> parents: 170 diff changeset	419
9b8ec700ede4 Clean up the weather data parser backend a bit. Matti Hamalainen <ccr@tnsp.org> parents: 170 diff changeset	420 ###
158 db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	421 ### Fetch FMI data
178 9b8ec700ede4 Clean up the weather data parser backend a bit. Matti Hamalainen <ccr@tnsp.org> parents: 170 diff changeset	422 ###
164 d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	423 if (opt_chk_bool("opt_fmi"))
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	424 {
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	425 die("FMI data scrape enabled, but no API key set.\n") unless opt_chk_valid("fmi_api_key", 10);
158 db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	426
189 3af8ea19a3e2 Improved weather fetching. Matti Hamalainen <ccr@tnsp.org> parents: 186 diff changeset	427 my $uri = "http://data.fmi.fi/fmi-apikey/".opt_get("fmi_api_key").
3af8ea19a3e2 Improved weather fetching. Matti Hamalainen <ccr@tnsp.org> parents: 186 diff changeset	428 "/wfs?request=getFeature&storedquery_id=fmi::observations::weather::".
3af8ea19a3e2 Improved weather fetching. Matti Hamalainen <ccr@tnsp.org> parents: 186 diff changeset	429 "multipointcoverage".
3af8ea19a3e2 Improved weather fetching. Matti Hamalainen <ccr@tnsp.org> parents: 186 diff changeset	430 # "timevaluepair".
194 55ed3db51ee7 Rename a function. Matti Hamalainen <ccr@tnsp.org> parents: 193 diff changeset	431 "&starttime=".format_time_gmt(time() - 10*60)."&endtime=".format_time_gmt(time()).
189 3af8ea19a3e2 Improved weather fetching. Matti Hamalainen <ccr@tnsp.org> parents: 186 diff changeset	432 "&parameters=temperature,humidity&maxlocations=50&bbox=19,59,32,70";
164 d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	433
189 3af8ea19a3e2 Improved weather fetching. Matti Hamalainen <ccr@tnsp.org> parents: 186 diff changeset	434 print STDERR "FMI URI: ".$uri."\n" if opt_chk_bool("debug");
3af8ea19a3e2 Improved weather fetching. Matti Hamalainen <ccr@tnsp.org> parents: 186 diff changeset	435
3af8ea19a3e2 Improved weather fetching. Matti Hamalainen <ccr@tnsp.org> parents: 186 diff changeset	436 my $res = fetch_http($uri);
164 d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	437 if ($res->code >= 200 && $res->code <= 201)
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	438 {
170 e4faa3746869 Remove useless parameters from XMLin(). Matti Hamalainen <ccr@tnsp.org> parents: 169 diff changeset	439 my $xml = XMLin($res->decoded_content);
169 c12e422a543a FMI data parsing kinda almost works. It is horrible, though. XML ... :Q Matti Hamalainen <ccr@tnsp.org> parents: 168 diff changeset	440 my $time_base = time();
189 3af8ea19a3e2 Improved weather fetching. Matti Hamalainen <ccr@tnsp.org> parents: 186 diff changeset	441
208 c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	442 if (defined($xml->{"wfs:member"}{"omso:GridSeriesObservation"}))
c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	443 {
c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	444 my $fdata = $xml->{"wfs:member"}{"omso:GridSeriesObservation"};
c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	445 my $fshit = $fdata->{"om:result"}{"gmlcov:MultiPointCoverage"};
c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	446 my @farray = ();
169 c12e422a543a FMI data parsing kinda almost works. It is horrible, though. XML ... :Q Matti Hamalainen <ccr@tnsp.org> parents: 168 diff changeset	447
208 c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	448 foreach my $fline (split(/\n/, $fshit->{"gml:domainSet"}{"gmlcov:SimpleMultiPoint"}{"gmlcov:positions"}))
169 c12e422a543a FMI data parsing kinda almost works. It is horrible, though. XML ... :Q Matti Hamalainen <ccr@tnsp.org> parents: 168 diff changeset	449 {
208 c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	450 if ($fline =~ /^\s([\+\-]?\d+\.\d)\s+([\+\-]?\d+\.\d)\s+(\d+)\s$/)
c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	451 {
c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	452 push(@farray, {"lat" => $1, "long" => $2, "time" => $3});
c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	453 }
169 c12e422a543a FMI data parsing kinda almost works. It is horrible, though. XML ... :Q Matti Hamalainen <ccr@tnsp.org> parents: 168 diff changeset	454 }
c12e422a543a FMI data parsing kinda almost works. It is horrible, though. XML ... :Q Matti Hamalainen <ccr@tnsp.org> parents: 168 diff changeset	455
208 c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	456 my $findex = 0;
c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	457 foreach my $fline (split(/\n/, $fshit->{"gml:rangeSet"}{"gml:DataBlock"}{"gml:doubleOrNilReasonTupleList"}))
169 c12e422a543a FMI data parsing kinda almost works. It is horrible, though. XML ... :Q Matti Hamalainen <ccr@tnsp.org> parents: 168 diff changeset	458 {
208 c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	459 if ($fline =~ /^\s([\+\-]?\d+\.\d\|NaN)\s+([\+\-]?\d+\.\d\|NaN)\s$/)
c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	460 {
c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	461 $farray[$findex]{"temp"} = $1 if ($1 ne "NaN");
c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	462 $farray[$findex]{"humidity"} = $2 if ($2 ne "NaN");
c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	463 $findex++;
c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	464 }
189 3af8ea19a3e2 Improved weather fetching. Matti Hamalainen <ccr@tnsp.org> parents: 186 diff changeset	465 }
208 c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	466 # XXX Hashify the array into lat/long keys
c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	467
c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	468 # This is horrible :S
c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	469 my $fcrap = $fdata->{"om:featureOfInterest"}{"sams:SF_SpatialSamplingFeature"}{"sams:shape"}{"gml:MultiPoint"}{"gml:pointMember"};
c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	470 foreach my $fnode (@{$fcrap})
189 3af8ea19a3e2 Improved weather fetching. Matti Hamalainen <ccr@tnsp.org> parents: 186 diff changeset	471 {
208 c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	472 my $floc = $fnode->{"gml:Point"};
c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	473 if ($floc->{"gml:pos"} =~ /^\s([\+\-]?\d+\.\d)\s+([\+\-]?\d+\.\d)\s$/)
189 3af8ea19a3e2 Improved weather fetching. Matti Hamalainen <ccr@tnsp.org> parents: 186 diff changeset	474 {
208 c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	475 my ($flat, $flong) = ($1, $2);
c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	476 # Should use a hash -
c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	477 foreach my $flol (@farray)
189 3af8ea19a3e2 Improved weather fetching. Matti Hamalainen <ccr@tnsp.org> parents: 186 diff changeset	478 {
208 c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	479 if ($flol->{"lat"} == $flat && $flol->{"long"} == $flong)
c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	480 {
c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	481 $weatherdata->{$floc->{"gml:name"}} =
c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	482 [
c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	483 1,
c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	484 $flol->{"time"},
c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	485 $flol->{"temp"},
c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	486 "",
c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	487 "",
c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	488 defined($flol->{"humidity"}) ? $flol->{"humidity"} : "",
c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	489 ];
c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	490 }
189 3af8ea19a3e2 Improved weather fetching. Matti Hamalainen <ccr@tnsp.org> parents: 186 diff changeset	491 }
3af8ea19a3e2 Improved weather fetching. Matti Hamalainen <ccr@tnsp.org> parents: 186 diff changeset	492 }
169 c12e422a543a FMI data parsing kinda almost works. It is horrible, though. XML ... :Q Matti Hamalainen <ccr@tnsp.org> parents: 168 diff changeset	493 }
208 c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	494 } else {
c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	495 # defined
c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	496 print STDERR "Invalid XML received:\n";
c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	497 print STDERR $res->decoded_content."\n\n";
169 c12e422a543a FMI data parsing kinda almost works. It is horrible, though. XML ... :Q Matti Hamalainen <ccr@tnsp.org> parents: 168 diff changeset	498 }
208 c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	499 } else {
c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	500 print STDERR "Error fetching FMI XML: ".$res->status_line."\n";
164 d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	501 }
158 db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	502 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	503
164 d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	504
178 9b8ec700ede4 Clean up the weather data parser backend a bit. Matti Hamalainen <ccr@tnsp.org> parents: 170 diff changeset	505 ###
158 db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	506 ### Output
178 9b8ec700ede4 Clean up the weather data parser backend a bit. Matti Hamalainen <ccr@tnsp.org> parents: 170 diff changeset	507 ###
164 d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	508 if (opt_chk_valid("outfile", 1)) {
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	509 open(STDOUT, '>', opt_get("outfile")) or die("Could not open output file '".opt_get("outfile")."'.\n");
158 db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	510 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	511
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	512 binmode STDOUT, ':encoding(utf-8)';
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	513
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	514 foreach my $key (sort { $a cmp $b } keys %$weatherdata)
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	515 {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	516 print STDOUT $key."\|".join("\|", @{$weatherdata->{$key}})."\n";
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	517 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	518
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	519 close(STDOUT);

Mercurial > hg > egg-tcls

annotate fetch_weather.pl @ 213:330d7e39c8f4