egg-tcls: fetch_weather.pl annotate

annotate fetch_weather.pl @ 438:a17b7fdd03c0

fetch_weather: Simplify and check for existence of output file before trying to read the old one.

author	Matti Hamalainen <ccr@tnsp.org>
date	Tue, 07 Feb 2017 10:35:11 +0200
parents	3c42ad35e157
children	19fe8ef0a902

rev	line source
381 846a0a95ad5a fetch_weather: Add back -w Matti Hamalainen <ccr@tnsp.org> parents: 380 diff changeset	1 #!/usr/bin/perl -w
182 24f9f38210fe Add comments. Matti Hamalainen <ccr@tnsp.org> parents: 178 diff changeset	2 ##########################################################################
24f9f38210fe Add comments. Matti Hamalainen <ccr@tnsp.org> parents: 178 diff changeset	3 #
407 b05da2077e44 fetch_weather: Bump copyright and version. Matti Hamalainen <ccr@tnsp.org> parents: 406 diff changeset	4 # Fetch Weather v0.8 by Matti 'ccr' Hamalainen <ccr@tnsp.org>
b05da2077e44 fetch_weather: Bump copyright and version. Matti Hamalainen <ccr@tnsp.org> parents: 406 diff changeset	5 # (C) Copyright 2014-2017 Tecnic Software productions (TNSP)
206 2ac661d551b3 Update documentation. Matti Hamalainen <ccr@tnsp.org> parents: 201 diff changeset	6 # This script is freely distributable under GNU GPL (version 2) license.
182 24f9f38210fe Add comments. Matti Hamalainen <ccr@tnsp.org> parents: 178 diff changeset	7 #
24f9f38210fe Add comments. Matti Hamalainen <ccr@tnsp.org> parents: 178 diff changeset	8 # Should be ran as a cronjob, and configured properly.
206 2ac661d551b3 Update documentation. Matti Hamalainen <ccr@tnsp.org> parents: 201 diff changeset	9 # /10 * * * perl -w /absolute/path/to/fetch_weather.pl /path/to/configfile
182 24f9f38210fe Add comments. Matti Hamalainen <ccr@tnsp.org> parents: 178 diff changeset	10 #
186 b41c525bdaf7 Add example configuration file. Matti Hamalainen <ccr@tnsp.org> parents: 185 diff changeset	11 # Configuration file example is in fetch_weather.config
182 24f9f38210fe Add comments. Matti Hamalainen <ccr@tnsp.org> parents: 178 diff changeset	12 #
206 2ac661d551b3 Update documentation. Matti Hamalainen <ccr@tnsp.org> parents: 201 diff changeset	13 # Requires various Perl modules, in Debian the packages should be:
2ac661d551b3 Update documentation. Matti Hamalainen <ccr@tnsp.org> parents: 201 diff changeset	14 # libwww-perl libxml-simple-perl libtimedate-perl
2ac661d551b3 Update documentation. Matti Hamalainen <ccr@tnsp.org> parents: 201 diff changeset	15 #
182 24f9f38210fe Add comments. Matti Hamalainen <ccr@tnsp.org> parents: 178 diff changeset	16 #
24f9f38210fe Add comments. Matti Hamalainen <ccr@tnsp.org> parents: 178 diff changeset	17 ##########################################################################
417 3c42ad35e157 fetch_weather: Cosmetic. Matti Hamalainen <ccr@tnsp.org> parents: 416 diff changeset	18 use 5.18.0;
158 db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	19 use strict;
380 729eb88d1c63 fetch_weather: "use warnings". Matti Hamalainen <ccr@tnsp.org> parents: 379 diff changeset	20 use warnings;
243 aef2cf1dc3be Use UTF8 source. Matti Hamalainen <ccr@tnsp.org> parents: 242 diff changeset	21 use utf8;
158 db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	22 use LWP::UserAgent;
399 a80de0130e45 fetch_weather: Support gzip encoded content in HTTP requests. Matti Hamalainen <ccr@tnsp.org> parents: 398 diff changeset	23 use HTTP::Message;
158 db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	24 use HTML::Entities;
399 a80de0130e45 fetch_weather: Support gzip encoded content in HTTP requests. Matti Hamalainen <ccr@tnsp.org> parents: 398 diff changeset	25 use Compress::Zlib;
158 db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	26 use XML::Simple;
189 3af8ea19a3e2 Improved weather fetching. Matti Hamalainen <ccr@tnsp.org> parents: 186 diff changeset	27 use Date::Format;
199 95012f8f687a Another round of timestamp parsing changes, urgh. Matti Hamalainen <ccr@tnsp.org> parents: 198 diff changeset	28 use Date::Parse;
158 db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	29 use Data::Dumper;
412 e0d86ccef873 fetch_weather: Change to using JSON static weather station metadata from digitraffic instead of the "old" API's CSV format data. Matti Hamalainen <ccr@tnsp.org> parents: 411 diff changeset	30 use File::Slurper qw(read_text write_binary);
e0d86ccef873 fetch_weather: Change to using JSON static weather station metadata from digitraffic instead of the "old" API's CSV format data. Matti Hamalainen <ccr@tnsp.org> parents: 411 diff changeset	31 use JSON;
166 c8d5f3a7c4b7 Add debug option. Matti Hamalainen <ccr@tnsp.org> parents: 164 diff changeset	32
400 f69aff75d1e2 fetch_weather: Cosmetic. Matti Hamalainen <ccr@tnsp.org> parents: 399 diff changeset	33
162 4f8a163b2bc1 Work on the backend weather data fetching. Matti Hamalainen <ccr@tnsp.org> parents: 158 diff changeset	34 ###
182 24f9f38210fe Add comments. Matti Hamalainen <ccr@tnsp.org> parents: 178 diff changeset	35 ### Configuration settings
162 4f8a163b2bc1 Work on the backend weather data fetching. Matti Hamalainen <ccr@tnsp.org> parents: 158 diff changeset	36 ###
164 d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	37 my %settings = (
166 c8d5f3a7c4b7 Add debug option. Matti Hamalainen <ccr@tnsp.org> parents: 164 diff changeset	38 "debug" => 0,
164 d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	39 "opt_fmi" => 0,
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	40 "opt_tiehallinto" => 0,
374 26b7963e7fbd fetch_weather: Oops, forgot to define the configuration setting Matti Hamalainen <ccr@tnsp.org> parents: 373 diff changeset	41 "purge_threshold" => 60,
164 d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	42 "fmi_api_key" => "",
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	43 "outfile" => "",
338 afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	44 "tiehallinto_static_meta" => "tiehallinto.meta",
185 64e06ee172c7 Add HTTP user agent setting. Matti Hamalainen <ccr@tnsp.org> parents: 182 diff changeset	45 "http_user_agent" => "Mozilla/4.0 (compatible; MSIE 6.0; MSIE 5.5; Windows NT 6.0) Opera 10.63 [en]",
164 d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	46 );
162 4f8a163b2bc1 Work on the backend weather data fetching. Matti Hamalainen <ccr@tnsp.org> parents: 158 diff changeset	47
178 9b8ec700ede4 Clean up the weather data parser backend a bit. Matti Hamalainen <ccr@tnsp.org> parents: 170 diff changeset	48
162 4f8a163b2bc1 Work on the backend weather data fetching. Matti Hamalainen <ccr@tnsp.org> parents: 158 diff changeset	49 ###
4f8a163b2bc1 Work on the backend weather data fetching. Matti Hamalainen <ccr@tnsp.org> parents: 158 diff changeset	50 ### Helper functions
4f8a163b2bc1 Work on the backend weather data fetching. Matti Hamalainen <ccr@tnsp.org> parents: 158 diff changeset	51 ###
164 d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	52 sub mlog($)
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	53 {
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	54 print STDERR $_[0];
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	55 }
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	56
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	57
178 9b8ec700ede4 Clean up the weather data parser backend a bit. Matti Hamalainen <ccr@tnsp.org> parents: 170 diff changeset	58 sub fetch_http($)
9b8ec700ede4 Clean up the weather data parser backend a bit. Matti Hamalainen <ccr@tnsp.org> parents: 170 diff changeset	59 {
198 60a4c69c7f25 Cosmetics. Matti Hamalainen <ccr@tnsp.org> parents: 197 diff changeset	60 my $agent = LWP::UserAgent->new;
60a4c69c7f25 Cosmetics. Matti Hamalainen <ccr@tnsp.org> parents: 197 diff changeset	61 $agent->agent($settings{"http_user_agent"});
208 c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	62 $agent->timeout(20);
178 9b8ec700ede4 Clean up the weather data parser backend a bit. Matti Hamalainen <ccr@tnsp.org> parents: 170 diff changeset	63
198 60a4c69c7f25 Cosmetics. Matti Hamalainen <ccr@tnsp.org> parents: 197 diff changeset	64 my $req = HTTP::Request->new(GET => $_[0]);
399 a80de0130e45 fetch_weather: Support gzip encoded content in HTTP requests. Matti Hamalainen <ccr@tnsp.org> parents: 398 diff changeset	65 $req->header('Accept-Encoding' => scalar HTTP::Message::decodable());
339 6ea410acbe74 Cosmetics. Matti Hamalainen <ccr@tnsp.org> parents: 338 diff changeset	66
411 fe47617e7251 fetch_weather: Add some debugging output to fetch_http(). Matti Hamalainen <ccr@tnsp.org> parents: 410 diff changeset	67 print STDERR "# FETCHING URL: ".$_[0]."\n" if (opt_get_int("debug") > 0);
fe47617e7251 fetch_weather: Add some debugging output to fetch_http(). Matti Hamalainen <ccr@tnsp.org> parents: 410 diff changeset	68
406 4c3abf052d01 fetch_weather: "use 5.018;" and some cleanups. Matti Hamalainen <ccr@tnsp.org> parents: 405 diff changeset	69 my $res = $agent->request($req);
411 fe47617e7251 fetch_weather: Add some debugging output to fetch_http(). Matti Hamalainen <ccr@tnsp.org> parents: 410 diff changeset	70
fe47617e7251 fetch_weather: Add some debugging output to fetch_http(). Matti Hamalainen <ccr@tnsp.org> parents: 410 diff changeset	71 if (opt_get_int("debug") > 0)
fe47617e7251 fetch_weather: Add some debugging output to fetch_http(). Matti Hamalainen <ccr@tnsp.org> parents: 410 diff changeset	72 {
fe47617e7251 fetch_weather: Add some debugging output to fetch_http(). Matti Hamalainen <ccr@tnsp.org> parents: 410 diff changeset	73 print STDERR "# Response: ".$res->code.": ".$res->message."\n";
fe47617e7251 fetch_weather: Add some debugging output to fetch_http(). Matti Hamalainen <ccr@tnsp.org> parents: 410 diff changeset	74 if ($res->code >= 200 && $res->code <= 201)
fe47617e7251 fetch_weather: Add some debugging output to fetch_http(). Matti Hamalainen <ccr@tnsp.org> parents: 410 diff changeset	75 {
fe47617e7251 fetch_weather: Add some debugging output to fetch_http(). Matti Hamalainen <ccr@tnsp.org> parents: 410 diff changeset	76 print STDERR
fe47617e7251 fetch_weather: Add some debugging output to fetch_http(). Matti Hamalainen <ccr@tnsp.org> parents: 410 diff changeset	77 "# Content-charset: ".$res->content_charset."\n".
fe47617e7251 fetch_weather: Add some debugging output to fetch_http(). Matti Hamalainen <ccr@tnsp.org> parents: 410 diff changeset	78 "# Content-encoding: ".$res->content_encoding."\n".
fe47617e7251 fetch_weather: Add some debugging output to fetch_http(). Matti Hamalainen <ccr@tnsp.org> parents: 410 diff changeset	79 "# Is decoded_content UTF8? ".(utf8::is_utf8($res->decoded_content) ? "yes" : "NO!")."\n";
fe47617e7251 fetch_weather: Add some debugging output to fetch_http(). Matti Hamalainen <ccr@tnsp.org> parents: 410 diff changeset	80 }
fe47617e7251 fetch_weather: Add some debugging output to fetch_http(). Matti Hamalainen <ccr@tnsp.org> parents: 410 diff changeset	81 }
fe47617e7251 fetch_weather: Add some debugging output to fetch_http(). Matti Hamalainen <ccr@tnsp.org> parents: 410 diff changeset	82
406 4c3abf052d01 fetch_weather: "use 5.018;" and some cleanups. Matti Hamalainen <ccr@tnsp.org> parents: 405 diff changeset	83 return $res;
178 9b8ec700ede4 Clean up the weather data parser backend a bit. Matti Hamalainen <ccr@tnsp.org> parents: 170 diff changeset	84 }
9b8ec700ede4 Clean up the weather data parser backend a bit. Matti Hamalainen <ccr@tnsp.org> parents: 170 diff changeset	85
9b8ec700ede4 Clean up the weather data parser backend a bit. Matti Hamalainen <ccr@tnsp.org> parents: 170 diff changeset	86
382 5573c1681da9 fetch_weather: Trim split input data lines in FMI XML. Matti Hamalainen <ccr@tnsp.org> parents: 381 diff changeset	87 sub str_trim($)
5573c1681da9 fetch_weather: Trim split input data lines in FMI XML. Matti Hamalainen <ccr@tnsp.org> parents: 381 diff changeset	88 {
5573c1681da9 fetch_weather: Trim split input data lines in FMI XML. Matti Hamalainen <ccr@tnsp.org> parents: 381 diff changeset	89 my $tmp = $_[0];
5573c1681da9 fetch_weather: Trim split input data lines in FMI XML. Matti Hamalainen <ccr@tnsp.org> parents: 381 diff changeset	90 $tmp =~ s/^\s*//;
5573c1681da9 fetch_weather: Trim split input data lines in FMI XML. Matti Hamalainen <ccr@tnsp.org> parents: 381 diff changeset	91 $tmp =~ s/\s*$//;
5573c1681da9 fetch_weather: Trim split input data lines in FMI XML. Matti Hamalainen <ccr@tnsp.org> parents: 381 diff changeset	92 return $tmp;
5573c1681da9 fetch_weather: Trim split input data lines in FMI XML. Matti Hamalainen <ccr@tnsp.org> parents: 381 diff changeset	93 }
5573c1681da9 fetch_weather: Trim split input data lines in FMI XML. Matti Hamalainen <ccr@tnsp.org> parents: 381 diff changeset	94
5573c1681da9 fetch_weather: Trim split input data lines in FMI XML. Matti Hamalainen <ccr@tnsp.org> parents: 381 diff changeset	95
178 9b8ec700ede4 Clean up the weather data parser backend a bit. Matti Hamalainen <ccr@tnsp.org> parents: 170 diff changeset	96 sub parse_timestamp($$)
9b8ec700ede4 Clean up the weather data parser backend a bit. Matti Hamalainen <ccr@tnsp.org> parents: 170 diff changeset	97 {
196 a38c123921a9 Change the timestamp parsing. Matti Hamalainen <ccr@tnsp.org> parents: 195 diff changeset	98 my ($str, $offs) = @_;
a38c123921a9 Change the timestamp parsing. Matti Hamalainen <ccr@tnsp.org> parents: 195 diff changeset	99 if ($str =~ /^(\d+):(\d+)$/)
a38c123921a9 Change the timestamp parsing. Matti Hamalainen <ccr@tnsp.org> parents: 195 diff changeset	100 {
a38c123921a9 Change the timestamp parsing. Matti Hamalainen <ccr@tnsp.org> parents: 195 diff changeset	101 return $offs + (60 * 60 * $1) + ($2 * 60);
a38c123921a9 Change the timestamp parsing. Matti Hamalainen <ccr@tnsp.org> parents: 195 diff changeset	102 }
a38c123921a9 Change the timestamp parsing. Matti Hamalainen <ccr@tnsp.org> parents: 195 diff changeset	103 else
a38c123921a9 Change the timestamp parsing. Matti Hamalainen <ccr@tnsp.org> parents: 195 diff changeset	104 {
a38c123921a9 Change the timestamp parsing. Matti Hamalainen <ccr@tnsp.org> parents: 195 diff changeset	105 return $offs;
a38c123921a9 Change the timestamp parsing. Matti Hamalainen <ccr@tnsp.org> parents: 195 diff changeset	106 }
178 9b8ec700ede4 Clean up the weather data parser backend a bit. Matti Hamalainen <ccr@tnsp.org> parents: 170 diff changeset	107 }
9b8ec700ede4 Clean up the weather data parser backend a bit. Matti Hamalainen <ccr@tnsp.org> parents: 170 diff changeset	108
9b8ec700ede4 Clean up the weather data parser backend a bit. Matti Hamalainen <ccr@tnsp.org> parents: 170 diff changeset	109
194 55ed3db51ee7 Rename a function. Matti Hamalainen <ccr@tnsp.org> parents: 193 diff changeset	110 sub format_time_gmt($)
189 3af8ea19a3e2 Improved weather fetching. Matti Hamalainen <ccr@tnsp.org> parents: 186 diff changeset	111 {
3af8ea19a3e2 Improved weather fetching. Matti Hamalainen <ccr@tnsp.org> parents: 186 diff changeset	112 # 2012-02-27T00:00:00Z
199 95012f8f687a Another round of timestamp parsing changes, urgh. Matti Hamalainen <ccr@tnsp.org> parents: 198 diff changeset	113 return time2str("%Y-%m-%dT%TZ", $_[0], "UTC");
189 3af8ea19a3e2 Improved weather fetching. Matti Hamalainen <ccr@tnsp.org> parents: 186 diff changeset	114 }
3af8ea19a3e2 Improved weather fetching. Matti Hamalainen <ccr@tnsp.org> parents: 186 diff changeset	115
193 93ec73deebc5 Cosmetics. Matti Hamalainen <ccr@tnsp.org> parents: 192 diff changeset	116
212 5f04ec60e8d9 weather: "Translate" rain state from Tiehallitus data. Matti Hamalainen <ccr@tnsp.org> parents: 208 diff changeset	117 my %th_rain_states =
5f04ec60e8d9 weather: "Translate" rain state from Tiehallitus data. Matti Hamalainen <ccr@tnsp.org> parents: 208 diff changeset	118 (
237 56af82fb7a6d Use lower case. Matti Hamalainen <ccr@tnsp.org> parents: 236 diff changeset	119 "Pouta" => "poutaa",
56af82fb7a6d Use lower case. Matti Hamalainen <ccr@tnsp.org> parents: 236 diff changeset	120 "Heikko" => "heikkoa sadetta",
56af82fb7a6d Use lower case. Matti Hamalainen <ccr@tnsp.org> parents: 236 diff changeset	121 "Kohtalainen" => "kohtalaista sadetta",
56af82fb7a6d Use lower case. Matti Hamalainen <ccr@tnsp.org> parents: 236 diff changeset	122 "Voimakas" => "voimakasta sadetta",
212 5f04ec60e8d9 weather: "Translate" rain state from Tiehallitus data. Matti Hamalainen <ccr@tnsp.org> parents: 208 diff changeset	123 );
5f04ec60e8d9 weather: "Translate" rain state from Tiehallitus data. Matti Hamalainen <ccr@tnsp.org> parents: 208 diff changeset	124
5f04ec60e8d9 weather: "Translate" rain state from Tiehallitus data. Matti Hamalainen <ccr@tnsp.org> parents: 208 diff changeset	125 my $th_rain_states_k = join("\|", map {quotemeta} sort { length($b)<=>length($a) } keys %th_rain_states);
5f04ec60e8d9 weather: "Translate" rain state from Tiehallitus data. Matti Hamalainen <ccr@tnsp.org> parents: 208 diff changeset	126
242 ffd49b583e34 Rename function. Matti Hamalainen <ccr@tnsp.org> parents: 238 diff changeset	127 sub translate_rain($)
212 5f04ec60e8d9 weather: "Translate" rain state from Tiehallitus data. Matti Hamalainen <ccr@tnsp.org> parents: 208 diff changeset	128 {
5f04ec60e8d9 weather: "Translate" rain state from Tiehallitus data. Matti Hamalainen <ccr@tnsp.org> parents: 208 diff changeset	129 my $tmp = $_[0];
237 56af82fb7a6d Use lower case. Matti Hamalainen <ccr@tnsp.org> parents: 236 diff changeset	130 $tmp =~ s/($th_rain_states_k)/$th_rain_states{$1}/igo;
212 5f04ec60e8d9 weather: "Translate" rain state from Tiehallitus data. Matti Hamalainen <ccr@tnsp.org> parents: 208 diff changeset	131 return $tmp;
5f04ec60e8d9 weather: "Translate" rain state from Tiehallitus data. Matti Hamalainen <ccr@tnsp.org> parents: 208 diff changeset	132 }
5f04ec60e8d9 weather: "Translate" rain state from Tiehallitus data. Matti Hamalainen <ccr@tnsp.org> parents: 208 diff changeset	133
331 b8ae5e99341b fetc_weather: Remove support for old HTML format parsing of Tiehallinto data, it's not provided in this format anymore. Matti Hamalainen <ccr@tnsp.org> parents: 325 diff changeset	134
244 7dea85b9a431 Translate cloudiness data. Matti Hamalainen <ccr@tnsp.org> parents: 243 diff changeset	135 my %th_cloud_states =
7dea85b9a431 Translate cloudiness data. Matti Hamalainen <ccr@tnsp.org> parents: 243 diff changeset	136 (
7dea85b9a431 Translate cloudiness data. Matti Hamalainen <ccr@tnsp.org> parents: 243 diff changeset	137 0 => "selkeää",
247 85ad2912e720 Possibly handle cloudyness translation better. Matti Hamalainen <ccr@tnsp.org> parents: 245 diff changeset	138 1 => "melkein selkeää",
85ad2912e720 Possibly handle cloudyness translation better. Matti Hamalainen <ccr@tnsp.org> parents: 245 diff changeset	139 2 => "verrattain selkeää",
85ad2912e720 Possibly handle cloudyness translation better. Matti Hamalainen <ccr@tnsp.org> parents: 245 diff changeset	140 3 => "verrattain selkeää",
85ad2912e720 Possibly handle cloudyness translation better. Matti Hamalainen <ccr@tnsp.org> parents: 245 diff changeset	141 4 => "puolipilvistä",
85ad2912e720 Possibly handle cloudyness translation better. Matti Hamalainen <ccr@tnsp.org> parents: 245 diff changeset	142 5 => "verrattain pilvistä",
85ad2912e720 Possibly handle cloudyness translation better. Matti Hamalainen <ccr@tnsp.org> parents: 245 diff changeset	143 6 => "verrattain pilvistä",
244 7dea85b9a431 Translate cloudiness data. Matti Hamalainen <ccr@tnsp.org> parents: 243 diff changeset	144 7 => "melkein pilvistä",
7dea85b9a431 Translate cloudiness data. Matti Hamalainen <ccr@tnsp.org> parents: 243 diff changeset	145 8 => "pilvistä",
7dea85b9a431 Translate cloudiness data. Matti Hamalainen <ccr@tnsp.org> parents: 243 diff changeset	146 );
7dea85b9a431 Translate cloudiness data. Matti Hamalainen <ccr@tnsp.org> parents: 243 diff changeset	147
7dea85b9a431 Translate cloudiness data. Matti Hamalainen <ccr@tnsp.org> parents: 243 diff changeset	148 sub translate_clouds($)
7dea85b9a431 Translate cloudiness data. Matti Hamalainen <ccr@tnsp.org> parents: 243 diff changeset	149 {
247 85ad2912e720 Possibly handle cloudyness translation better. Matti Hamalainen <ccr@tnsp.org> parents: 245 diff changeset	150 return "" if ($_[0] eq "NaN" \|\| $_[0] eq "");
85ad2912e720 Possibly handle cloudyness translation better. Matti Hamalainen <ccr@tnsp.org> parents: 245 diff changeset	151 my $tmp = int($_[0]);
244 7dea85b9a431 Translate cloudiness data. Matti Hamalainen <ccr@tnsp.org> parents: 243 diff changeset	152 foreach my $n (sort { $a <=> $b } keys %th_cloud_states)
7dea85b9a431 Translate cloudiness data. Matti Hamalainen <ccr@tnsp.org> parents: 243 diff changeset	153 {
247 85ad2912e720 Possibly handle cloudyness translation better. Matti Hamalainen <ccr@tnsp.org> parents: 245 diff changeset	154 return $th_cloud_states{$n}." (".$n."/8)" if ($tmp == $n);
244 7dea85b9a431 Translate cloudiness data. Matti Hamalainen <ccr@tnsp.org> parents: 243 diff changeset	155 }
7dea85b9a431 Translate cloudiness data. Matti Hamalainen <ccr@tnsp.org> parents: 243 diff changeset	156 return $tmp;
7dea85b9a431 Translate cloudiness data. Matti Hamalainen <ccr@tnsp.org> parents: 243 diff changeset	157 }
7dea85b9a431 Translate cloudiness data. Matti Hamalainen <ccr@tnsp.org> parents: 243 diff changeset	158
212 5f04ec60e8d9 weather: "Translate" rain state from Tiehallitus data. Matti Hamalainen <ccr@tnsp.org> parents: 208 diff changeset	159
410 5c97006c89e6 fetch_weather: Cleanups. Matti Hamalainen <ccr@tnsp.org> parents: 407 diff changeset	160 ### Return either data or if not defined, empty string
228 fe420576f284 The kludges intensify. Matti Hamalainen <ccr@tnsp.org> parents: 213 diff changeset	161 sub plonk_data($)
fe420576f284 The kludges intensify. Matti Hamalainen <ccr@tnsp.org> parents: 213 diff changeset	162 {
fe420576f284 The kludges intensify. Matti Hamalainen <ccr@tnsp.org> parents: 213 diff changeset	163 return defined($_[0]) ? $_[0] : "";
fe420576f284 The kludges intensify. Matti Hamalainen <ccr@tnsp.org> parents: 213 diff changeset	164 }
fe420576f284 The kludges intensify. Matti Hamalainen <ccr@tnsp.org> parents: 213 diff changeset	165
fe420576f284 The kludges intensify. Matti Hamalainen <ccr@tnsp.org> parents: 213 diff changeset	166
410 5c97006c89e6 fetch_weather: Cleanups. Matti Hamalainen <ccr@tnsp.org> parents: 407 diff changeset	167 ### Same as plonk_data() but also lowercase the data string
237 56af82fb7a6d Use lower case. Matti Hamalainen <ccr@tnsp.org> parents: 236 diff changeset	168 sub plonk_data_lc($)
56af82fb7a6d Use lower case. Matti Hamalainen <ccr@tnsp.org> parents: 236 diff changeset	169 {
56af82fb7a6d Use lower case. Matti Hamalainen <ccr@tnsp.org> parents: 236 diff changeset	170 return defined($_[0]) ? lc($_[0]) : "";
56af82fb7a6d Use lower case. Matti Hamalainen <ccr@tnsp.org> parents: 236 diff changeset	171 }
56af82fb7a6d Use lower case. Matti Hamalainen <ccr@tnsp.org> parents: 236 diff changeset	172
56af82fb7a6d Use lower case. Matti Hamalainen <ccr@tnsp.org> parents: 236 diff changeset	173
158 db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	174 ###
178 9b8ec700ede4 Clean up the weather data parser backend a bit. Matti Hamalainen <ccr@tnsp.org> parents: 170 diff changeset	175 ### Configuration handling
164 d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	176 ###
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	177 sub opt_chk_bool($)
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	178 {
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	179 if (defined($settings{$_[0]}))
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	180 {
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	181 my $val = $settings{$_[0]};
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	182 return ($val == 1 \|\| $val eq "true" \|\| $val eq "on" \|\| $val eq "1");
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	183 }
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	184 else
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	185 {
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	186 return 0;
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	187 }
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	188 }
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	189
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	190
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	191 sub opt_chk_valid($$)
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	192 {
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	193 if (defined($settings{$_[0]}))
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	194 {
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	195 my $val = $settings{$_[0]};
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	196 return length($val) >= $_[1];
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	197 }
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	198 else
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	199 {
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	200 return 0;
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	201 }
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	202 }
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	203
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	204
324 19c64798de91 weather: Add some better optional debugging. Matti Hamalainen <ccr@tnsp.org> parents: 247 diff changeset	205 sub opt_get_int($)
19c64798de91 weather: Add some better optional debugging. Matti Hamalainen <ccr@tnsp.org> parents: 247 diff changeset	206 {
19c64798de91 weather: Add some better optional debugging. Matti Hamalainen <ccr@tnsp.org> parents: 247 diff changeset	207 if (defined($settings{$_[0]}))
19c64798de91 weather: Add some better optional debugging. Matti Hamalainen <ccr@tnsp.org> parents: 247 diff changeset	208 {
19c64798de91 weather: Add some better optional debugging. Matti Hamalainen <ccr@tnsp.org> parents: 247 diff changeset	209 return int($settings{$_[0]});
19c64798de91 weather: Add some better optional debugging. Matti Hamalainen <ccr@tnsp.org> parents: 247 diff changeset	210 }
19c64798de91 weather: Add some better optional debugging. Matti Hamalainen <ccr@tnsp.org> parents: 247 diff changeset	211 else
19c64798de91 weather: Add some better optional debugging. Matti Hamalainen <ccr@tnsp.org> parents: 247 diff changeset	212 {
19c64798de91 weather: Add some better optional debugging. Matti Hamalainen <ccr@tnsp.org> parents: 247 diff changeset	213 return -1;
19c64798de91 weather: Add some better optional debugging. Matti Hamalainen <ccr@tnsp.org> parents: 247 diff changeset	214 }
19c64798de91 weather: Add some better optional debugging. Matti Hamalainen <ccr@tnsp.org> parents: 247 diff changeset	215 }
19c64798de91 weather: Add some better optional debugging. Matti Hamalainen <ccr@tnsp.org> parents: 247 diff changeset	216
19c64798de91 weather: Add some better optional debugging. Matti Hamalainen <ccr@tnsp.org> parents: 247 diff changeset	217
164 d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	218 sub opt_get($)
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	219 {
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	220 if (defined($settings{$_[0]}))
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	221 {
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	222 return $settings{$_[0]};
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	223 }
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	224 else
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	225 {
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	226 return undef;
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	227 }
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	228 }
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	229
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	230
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	231 sub opt_read_config($)
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	232 {
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	233 my $filename = $_[0];
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	234 my $errors = 0;
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	235 my $line = 0;
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	236
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	237 open(CONFFILE, "<", $filename) or die("Could not open configuration '".$filename."'!\n");
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	238 while (<CONFFILE>)
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	239 {
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	240 $line++;
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	241 chomp;
353 a52a983c0c14 Cosmetics. Matti Hamalainen <ccr@tnsp.org> parents: 352 diff changeset	242 if (/(^\s#\|^\s$)/)
a52a983c0c14 Cosmetics. Matti Hamalainen <ccr@tnsp.org> parents: 352 diff changeset	243 {
164 d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	244 # Ignore comments and empty lines
353 a52a983c0c14 Cosmetics. Matti Hamalainen <ccr@tnsp.org> parents: 352 diff changeset	245 }
386 299744a6c592 Cosmetics. Matti Hamalainen <ccr@tnsp.org> parents: 385 diff changeset	246 elsif (/^\s\"?([a-zA-Z0-9_]+)\"?\s=>?\s(\d+),?\s$/)
299744a6c592 Cosmetics. Matti Hamalainen <ccr@tnsp.org> parents: 385 diff changeset	247 {
164 d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	248 my $key = lc($1);
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	249 my $value = $2;
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	250 if (defined($settings{$key})) {
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	251 $settings{$key} = $value;
353 a52a983c0c14 Cosmetics. Matti Hamalainen <ccr@tnsp.org> parents: 352 diff changeset	252 }
a52a983c0c14 Cosmetics. Matti Hamalainen <ccr@tnsp.org> parents: 352 diff changeset	253 else
a52a983c0c14 Cosmetics. Matti Hamalainen <ccr@tnsp.org> parents: 352 diff changeset	254 {
164 d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	255 mlog("[$filename:$line] Unknown setting '$key' = $value\n");
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	256 $errors = 1;
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	257 }
353 a52a983c0c14 Cosmetics. Matti Hamalainen <ccr@tnsp.org> parents: 352 diff changeset	258 }
386 299744a6c592 Cosmetics. Matti Hamalainen <ccr@tnsp.org> parents: 385 diff changeset	259 elsif (/^\s\"?([a-zA-Z0-9_]+)\"?\s=>?\s\"(.?)\",?\s*$/)
299744a6c592 Cosmetics. Matti Hamalainen <ccr@tnsp.org> parents: 385 diff changeset	260 {
164 d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	261 my $key = lc($1);
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	262 my $value = $2;
353 a52a983c0c14 Cosmetics. Matti Hamalainen <ccr@tnsp.org> parents: 352 diff changeset	263 if (defined($settings{$key}))
a52a983c0c14 Cosmetics. Matti Hamalainen <ccr@tnsp.org> parents: 352 diff changeset	264 {
164 d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	265 $settings{$key} = $value;
353 a52a983c0c14 Cosmetics. Matti Hamalainen <ccr@tnsp.org> parents: 352 diff changeset	266 }
a52a983c0c14 Cosmetics. Matti Hamalainen <ccr@tnsp.org> parents: 352 diff changeset	267 else
a52a983c0c14 Cosmetics. Matti Hamalainen <ccr@tnsp.org> parents: 352 diff changeset	268 {
164 d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	269 mlog("[$filename:$line] Unknown setting '$key' = '$value'\n");
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	270 $errors = 1;
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	271 }
353 a52a983c0c14 Cosmetics. Matti Hamalainen <ccr@tnsp.org> parents: 352 diff changeset	272 }
a52a983c0c14 Cosmetics. Matti Hamalainen <ccr@tnsp.org> parents: 352 diff changeset	273 else
a52a983c0c14 Cosmetics. Matti Hamalainen <ccr@tnsp.org> parents: 352 diff changeset	274 {
164 d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	275 mlog("[$filename:$line] Syntax error: $_\n");
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	276 $errors = 1;
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	277 }
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	278 }
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	279 close(CONFFILE);
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	280 return $errors;
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	281 }
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	282
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	283
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	284 ###
158 db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	285 ### Main program begins
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	286 ###
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	287 my $weatherdata = {};
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	288
164 d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	289 die(
407 b05da2077e44 fetch_weather: Bump copyright and version. Matti Hamalainen <ccr@tnsp.org> parents: 406 diff changeset	290 "Weather Fetch v0.8 by ccr/TNSP <ccr\@tnsp.org>\n".
404 32be5d1dca29 fetch_weather: Add commandline option for forcing update of static metadata caches. Matti Hamalainen <ccr@tnsp.org> parents: 403 diff changeset	291 "Usage: $0 <config file> [force]\n"
164 d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	292 ) unless scalar(@ARGV) >= 1;
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	293
178 9b8ec700ede4 Clean up the weather data parser backend a bit. Matti Hamalainen <ccr@tnsp.org> parents: 170 diff changeset	294 my $cfgfile = shift;
9b8ec700ede4 Clean up the weather data parser backend a bit. Matti Hamalainen <ccr@tnsp.org> parents: 170 diff changeset	295 opt_read_config($cfgfile) == 0 or die("Errors while parsing configuration file '".$cfgfile."'.\n");
405 26c613434971 fetch_weather: Better check for 'force' commandline parameter. Matti Hamalainen <ccr@tnsp.org> parents: 404 diff changeset	296 my $force_update = scalar(@ARGV) >= 1 && (shift eq "force");
164 d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	297
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	298
178 9b8ec700ede4 Clean up the weather data parser backend a bit. Matti Hamalainen <ccr@tnsp.org> parents: 170 diff changeset	299 ###
354 de411e7834aa fetch_weather: Reload old data from stored data. Matti Hamalainen <ccr@tnsp.org> parents: 353 diff changeset	300 ### Load already cached data
de411e7834aa fetch_weather: Reload old data from stored data. Matti Hamalainen <ccr@tnsp.org> parents: 353 diff changeset	301 ###
de411e7834aa fetch_weather: Reload old data from stored data. Matti Hamalainen <ccr@tnsp.org> parents: 353 diff changeset	302 if (opt_chk_valid("outfile", 1))
de411e7834aa fetch_weather: Reload old data from stored data. Matti Hamalainen <ccr@tnsp.org> parents: 353 diff changeset	303 {
438 a17b7fdd03c0 fetch_weather: Simplify and check for existence of output file before trying to read the old one. Matti Hamalainen <ccr@tnsp.org> parents: 417 diff changeset	304 my $filename = opt_get("outfile");
a17b7fdd03c0 fetch_weather: Simplify and check for existence of output file before trying to read the old one. Matti Hamalainen <ccr@tnsp.org> parents: 417 diff changeset	305 if (-e "$filename")
354 de411e7834aa fetch_weather: Reload old data from stored data. Matti Hamalainen <ccr@tnsp.org> parents: 353 diff changeset	306 {
438 a17b7fdd03c0 fetch_weather: Simplify and check for existence of output file before trying to read the old one. Matti Hamalainen <ccr@tnsp.org> parents: 417 diff changeset	307 my $str = read_text($filename);
a17b7fdd03c0 fetch_weather: Simplify and check for existence of output file before trying to read the old one. Matti Hamalainen <ccr@tnsp.org> parents: 417 diff changeset	308 if (defined($str))
354 de411e7834aa fetch_weather: Reload old data from stored data. Matti Hamalainen <ccr@tnsp.org> parents: 353 diff changeset	309 {
438 a17b7fdd03c0 fetch_weather: Simplify and check for existence of output file before trying to read the old one. Matti Hamalainen <ccr@tnsp.org> parents: 417 diff changeset	310 foreach my $line (split(/\s\n\s/, $str))
354 de411e7834aa fetch_weather: Reload old data from stored data. Matti Hamalainen <ccr@tnsp.org> parents: 353 diff changeset	311 {
438 a17b7fdd03c0 fetch_weather: Simplify and check for existence of output file before trying to read the old one. Matti Hamalainen <ccr@tnsp.org> parents: 417 diff changeset	312 my @mtmp = split(/\\|/, $line, -1);
a17b7fdd03c0 fetch_weather: Simplify and check for existence of output file before trying to read the old one. Matti Hamalainen <ccr@tnsp.org> parents: 417 diff changeset	313 if (scalar(\@mtmp) >= 3)
a17b7fdd03c0 fetch_weather: Simplify and check for existence of output file before trying to read the old one. Matti Hamalainen <ccr@tnsp.org> parents: 417 diff changeset	314 {
a17b7fdd03c0 fetch_weather: Simplify and check for existence of output file before trying to read the old one. Matti Hamalainen <ccr@tnsp.org> parents: 417 diff changeset	315 $weatherdata->{shift @mtmp} = \@mtmp;
a17b7fdd03c0 fetch_weather: Simplify and check for existence of output file before trying to read the old one. Matti Hamalainen <ccr@tnsp.org> parents: 417 diff changeset	316 }
354 de411e7834aa fetch_weather: Reload old data from stored data. Matti Hamalainen <ccr@tnsp.org> parents: 353 diff changeset	317 }
438 a17b7fdd03c0 fetch_weather: Simplify and check for existence of output file before trying to read the old one. Matti Hamalainen <ccr@tnsp.org> parents: 417 diff changeset	318 print STDERR scalar(keys %$weatherdata)." old records reloaded.\n" if (opt_get_int("debug") > 0);
354 de411e7834aa fetch_weather: Reload old data from stored data. Matti Hamalainen <ccr@tnsp.org> parents: 353 diff changeset	319 }
de411e7834aa fetch_weather: Reload old data from stored data. Matti Hamalainen <ccr@tnsp.org> parents: 353 diff changeset	320 }
de411e7834aa fetch_weather: Reload old data from stored data. Matti Hamalainen <ccr@tnsp.org> parents: 353 diff changeset	321 }
de411e7834aa fetch_weather: Reload old data from stored data. Matti Hamalainen <ccr@tnsp.org> parents: 353 diff changeset	322
de411e7834aa fetch_weather: Reload old data from stored data. Matti Hamalainen <ccr@tnsp.org> parents: 353 diff changeset	323
de411e7834aa fetch_weather: Reload old data from stored data. Matti Hamalainen <ccr@tnsp.org> parents: 353 diff changeset	324 ###
338 afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	325 ### Fetch Tiehallinto data
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	326 ###
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	327 if (opt_chk_bool("opt_tiehallinto"))
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	328 {
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	329 my $uri = "http://tie.digitraffic.fi/sujuvuus/ws/roadWeather";
406 4c3abf052d01 fetch_weather: "use 5.018;" and some cleanups. Matti Hamalainen <ccr@tnsp.org> parents: 405 diff changeset	330 print STDERR "Fetching Tiehallinto road weather data from ".$uri."\n" if (opt_get_int("debug") > 0);
338 afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	331 my $res = fetch_http($uri);
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	332 if ($res->code >= 200 && $res->code <= 201)
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	333 {
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	334 my $xml = XMLin($res->decoded_content);
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	335
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	336 if (!defined($xml->{"soap:Body"}) \|\| !defined($xml->{"soap:Body"}{"RoadWeatherResponse"}))
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	337 {
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	338 print STDERR "ERROR: SOAP call result did not contain required data.\n";
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	339 print STDERR $res->decoded_content."\n\n";
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	340 }
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	341 else
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	342 {
406 4c3abf052d01 fetch_weather: "use 5.018;" and some cleanups. Matti Hamalainen <ccr@tnsp.org> parents: 405 diff changeset	343 # Parse the XML
338 afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	344 my $data = $xml->{"soap:Body"}{"RoadWeatherResponse"};
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	345
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	346 # Check if we need to update the static meta data
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	347 my $meta_file = opt_get("tiehallinto_static_meta");
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	348 my $fetch_meta = (-e $meta_file) ? 0 : 1;
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	349
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	350 if (defined($data->{"laststaticdataupdate"}))
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	351 {
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	352 # Compare metadata cache file modification timestamp to info in XML
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	353 my $tmp1 = str2time($data->{"laststaticdataupdate"});
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	354 my $tmp2 = (-e $meta_file) ? (stat($meta_file))[9] : -1;
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	355 $fetch_meta = 1 unless ($tmp1 < $tmp2);
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	356 }
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	357
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	358 # Fetch or read the cache
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	359 my $meta_str;
404 32be5d1dca29 fetch_weather: Add commandline option for forcing update of static metadata caches. Matti Hamalainen <ccr@tnsp.org> parents: 403 diff changeset	360 if ($fetch_meta \|\| $force_update)
338 afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	361 {
412 e0d86ccef873 fetch_weather: Change to using JSON static weather station metadata from digitraffic instead of the "old" API's CSV format data. Matti Hamalainen <ccr@tnsp.org> parents: 411 diff changeset	362 my $uri = "http://tie.digitraffic.fi/api/v1/metadata/weather-stations";
406 4c3abf052d01 fetch_weather: "use 5.018;" and some cleanups. Matti Hamalainen <ccr@tnsp.org> parents: 405 diff changeset	363 print STDERR "Fetching Tiehallinto static meta data from $uri\n" if (opt_get_int("debug") > 1);
338 afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	364 my $res = fetch_http($uri);
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	365 die("Failed to fetch $uri data.\n") unless ($res->code <= 200 && $res->code <= 201);
339 6ea410acbe74 Cosmetics. Matti Hamalainen <ccr@tnsp.org> parents: 338 diff changeset	366
338 afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	367 $meta_str = $res->decoded_content;
413 b63525aed190 fetch_weather: Unfortunately we have to deal with some UTF-8 brokenness here. Matti Hamalainen <ccr@tnsp.org> parents: 412 diff changeset	368
b63525aed190 fetch_weather: Unfortunately we have to deal with some UTF-8 brokenness here. Matti Hamalainen <ccr@tnsp.org> parents: 412 diff changeset	369 # XXX: This is a hack. For some reason the data does not get utf8 flagged internally.
b63525aed190 fetch_weather: Unfortunately we have to deal with some UTF-8 brokenness here. Matti Hamalainen <ccr@tnsp.org> parents: 412 diff changeset	370 if (!utf8::is_utf8($meta_str))
b63525aed190 fetch_weather: Unfortunately we have to deal with some UTF-8 brokenness here. Matti Hamalainen <ccr@tnsp.org> parents: 412 diff changeset	371 {
b63525aed190 fetch_weather: Unfortunately we have to deal with some UTF-8 brokenness here. Matti Hamalainen <ccr@tnsp.org> parents: 412 diff changeset	372 printf STDERR "Upgrading meta_str to UTF-8.\n" if (opt_get_int("debug") > 0);
b63525aed190 fetch_weather: Unfortunately we have to deal with some UTF-8 brokenness here. Matti Hamalainen <ccr@tnsp.org> parents: 412 diff changeset	373 utf8::upgrade($meta_str);
b63525aed190 fetch_weather: Unfortunately we have to deal with some UTF-8 brokenness here. Matti Hamalainen <ccr@tnsp.org> parents: 412 diff changeset	374 }
b63525aed190 fetch_weather: Unfortunately we have to deal with some UTF-8 brokenness here. Matti Hamalainen <ccr@tnsp.org> parents: 412 diff changeset	375
412 e0d86ccef873 fetch_weather: Change to using JSON static weather station metadata from digitraffic instead of the "old" API's CSV format data. Matti Hamalainen <ccr@tnsp.org> parents: 411 diff changeset	376 $fetch_meta = 1;
338 afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	377 }
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	378 else
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	379 {
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	380 print STDERR "Using CACHED Tiehallinto static meta data from '$meta_file'.\n" if (opt_get_int("debug") > 0);
416 c88f7449b526 fetch_weather: Possibly another UTF8 fix :S Matti Hamalainen <ccr@tnsp.org> parents: 414 diff changeset	381 $meta_str = read_text($meta_file);
338 afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	382 }
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	383
416 c88f7449b526 fetch_weather: Possibly another UTF8 fix :S Matti Hamalainen <ccr@tnsp.org> parents: 414 diff changeset	384 print STDERR "Is meta_str UTF8? ".(utf8::is_utf8($meta_str) ? "yes" : "NO!")."\n" if (opt_get_int("debug") > 0);
c88f7449b526 fetch_weather: Possibly another UTF8 fix :S Matti Hamalainen <ccr@tnsp.org> parents: 414 diff changeset	385
338 afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	386 # Parse the data ..
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	387 my $meta_data = {};
412 e0d86ccef873 fetch_weather: Change to using JSON static weather station metadata from digitraffic instead of the "old" API's CSV format data. Matti Hamalainen <ccr@tnsp.org> parents: 411 diff changeset	388 my $json = JSON->new->decode($meta_str);
398 6419327df9e7 fetch_weather: Cosmetic. Matti Hamalainen <ccr@tnsp.org> parents: 397 diff changeset	389
412 e0d86ccef873 fetch_weather: Change to using JSON static weather station metadata from digitraffic instead of the "old" API's CSV format data. Matti Hamalainen <ccr@tnsp.org> parents: 411 diff changeset	390 if ($fetch_meta)
338 afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	391 {
412 e0d86ccef873 fetch_weather: Change to using JSON static weather station metadata from digitraffic instead of the "old" API's CSV format data. Matti Hamalainen <ccr@tnsp.org> parents: 411 diff changeset	392 # Save new cache, in more optimal form, if needed.
e0d86ccef873 fetch_weather: Change to using JSON static weather station metadata from digitraffic instead of the "old" API's CSV format data. Matti Hamalainen <ccr@tnsp.org> parents: 411 diff changeset	393 print STDERR "Storing to cache '$meta_file'.\n" if (opt_get_int("debug") > 0);
e0d86ccef873 fetch_weather: Change to using JSON static weather station metadata from digitraffic instead of the "old" API's CSV format data. Matti Hamalainen <ccr@tnsp.org> parents: 411 diff changeset	394 write_binary($meta_file, JSON->new->encode($json));
e0d86ccef873 fetch_weather: Change to using JSON static weather station metadata from digitraffic instead of the "old" API's CSV format data. Matti Hamalainen <ccr@tnsp.org> parents: 411 diff changeset	395 }
e0d86ccef873 fetch_weather: Change to using JSON static weather station metadata from digitraffic instead of the "old" API's CSV format data. Matti Hamalainen <ccr@tnsp.org> parents: 411 diff changeset	396
e0d86ccef873 fetch_weather: Change to using JSON static weather station metadata from digitraffic instead of the "old" API's CSV format data. Matti Hamalainen <ccr@tnsp.org> parents: 411 diff changeset	397 foreach my $ms (@{$json->{"features"}})
e0d86ccef873 fetch_weather: Change to using JSON static weather station metadata from digitraffic instead of the "old" API's CSV format data. Matti Hamalainen <ccr@tnsp.org> parents: 411 diff changeset	398 {
e0d86ccef873 fetch_weather: Change to using JSON static weather station metadata from digitraffic instead of the "old" API's CSV format data. Matti Hamalainen <ccr@tnsp.org> parents: 411 diff changeset	399 if (defined($ms->{"properties"}) &&
e0d86ccef873 fetch_weather: Change to using JSON static weather station metadata from digitraffic instead of the "old" API's CSV format data. Matti Hamalainen <ccr@tnsp.org> parents: 411 diff changeset	400 defined($ms->{"geometry"}{"coordinates"}) &&
e0d86ccef873 fetch_weather: Change to using JSON static weather station metadata from digitraffic instead of the "old" API's CSV format data. Matti Hamalainen <ccr@tnsp.org> parents: 411 diff changeset	401 defined($ms->{"properties"}{"names"}{"fi"}))
338 afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	402 {
412 e0d86ccef873 fetch_weather: Change to using JSON static weather station metadata from digitraffic instead of the "old" API's CSV format data. Matti Hamalainen <ccr@tnsp.org> parents: 411 diff changeset	403 $meta_data->{$ms->{"id"}} = $ms;
338 afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	404 }
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	405 }
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	406
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	407 # Parse XML and combine with the station meta data
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	408 if (defined($data->{"roadweatherdata"}))
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	409 {
349 f7793ef7c925 fetch_weather: Improve debugging. Matti Hamalainen <ccr@tnsp.org> parents: 343 diff changeset	410 my $nrecords = 0;
338 afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	411 foreach my $wdata (@{$data->{"roadweatherdata"}{"roadweather"}})
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	412 {
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	413 my $wid = $wdata->{"stationid"};
412 e0d86ccef873 fetch_weather: Change to using JSON static weather station metadata from digitraffic instead of the "old" API's CSV format data. Matti Hamalainen <ccr@tnsp.org> parents: 411 diff changeset	414 if (defined($meta_data->{$wid}))
338 afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	415 {
349 f7793ef7c925 fetch_weather: Improve debugging. Matti Hamalainen <ccr@tnsp.org> parents: 343 diff changeset	416 $nrecords++;
412 e0d86ccef873 fetch_weather: Change to using JSON static weather station metadata from digitraffic instead of the "old" API's CSV format data. Matti Hamalainen <ccr@tnsp.org> parents: 411 diff changeset	417 $weatherdata->{$meta_data->{$wid}{"properties"}{"names"}{"fi"}} =
338 afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	418 [
343 3fff571a904b fetch_weather: Some fixes to make this actually work. Matti Hamalainen <ccr@tnsp.org> parents: 340 diff changeset	419 1,
414 d623652df6b5 fetch_weather / weather: Add weather measurement station location data (GPS WGS-84 lat/long/height) to weather cache. Not used in weather.tcl yet, tho. Matti Hamalainen <ccr@tnsp.org> parents: 413 diff changeset	420
d623652df6b5 fetch_weather / weather: Add weather measurement station location data (GPS WGS-84 lat/long/height) to weather cache. Not used in weather.tcl yet, tho. Matti Hamalainen <ccr@tnsp.org> parents: 413 diff changeset	421 $meta_data->{$wid}{"geometry"}{"coordinates"}[1],
d623652df6b5 fetch_weather / weather: Add weather measurement station location data (GPS WGS-84 lat/long/height) to weather cache. Not used in weather.tcl yet, tho. Matti Hamalainen <ccr@tnsp.org> parents: 413 diff changeset	422 $meta_data->{$wid}{"geometry"}{"coordinates"}[0],
d623652df6b5 fetch_weather / weather: Add weather measurement station location data (GPS WGS-84 lat/long/height) to weather cache. Not used in weather.tcl yet, tho. Matti Hamalainen <ccr@tnsp.org> parents: 413 diff changeset	423 $meta_data->{$wid}{"geometry"}{"coordinates"}[2],
d623652df6b5 fetch_weather / weather: Add weather measurement station location data (GPS WGS-84 lat/long/height) to weather cache. Not used in weather.tcl yet, tho. Matti Hamalainen <ccr@tnsp.org> parents: 413 diff changeset	424
338 afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	425 str2time(plonk_data($wdata->{"measurementtime"}{"utc"})),
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	426 plonk_data($wdata->{"airtemperature1"}),
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	427
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	428 plonk_data($wdata->{"humidity"}),
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	429 plonk_data($wdata->{"averagewindspeed"}),
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	430 ];
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	431 }
377 0a6d9f0f0a57 fetch_weather: Fix Tiehallinto metadata handling. Matti Hamalainen <ccr@tnsp.org> parents: 375 diff changeset	432 else
0a6d9f0f0a57 fetch_weather: Fix Tiehallinto metadata handling. Matti Hamalainen <ccr@tnsp.org> parents: 375 diff changeset	433 {
397 799320c3386e Change "Station ID #" not defined?" to a debug message. Matti Hamalainen <ccr@tnsp.org> parents: 396 diff changeset	434 print STDERR "Station ID #".$wid." not defined?\n" if (opt_get_int("debug") > 0);
400 f69aff75d1e2 fetch_weather: Cosmetic. Matti Hamalainen <ccr@tnsp.org> parents: 399 diff changeset	435 #.Dumper($meta_data->{$wid});
377 0a6d9f0f0a57 fetch_weather: Fix Tiehallinto metadata handling. Matti Hamalainen <ccr@tnsp.org> parents: 375 diff changeset	436 }
338 afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	437 }
349 f7793ef7c925 fetch_weather: Improve debugging. Matti Hamalainen <ccr@tnsp.org> parents: 343 diff changeset	438 print STDERR $nrecords." records from Tiehallinto.\n" if (opt_get_int("debug") > 0);
338 afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	439 }
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	440 else
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	441 {
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	442 print STDERR "ERROR: Invalid (or unsupported) road weather data blob.\n";
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	443 print STDERR $res->decoded_content."\n\n";
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	444 }
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	445 }
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	446 }
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	447 }
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	448
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	449
afb8c0e2d513 fetch_weather: Implement support for new Tiehallinto data API. Matti Hamalainen <ccr@tnsp.org> parents: 332 diff changeset	450 ###
158 db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	451 ### Fetch FMI data
178 9b8ec700ede4 Clean up the weather data parser backend a bit. Matti Hamalainen <ccr@tnsp.org> parents: 170 diff changeset	452 ###
164 d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	453 if (opt_chk_bool("opt_fmi"))
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	454 {
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	455 die("FMI data scrape enabled, but no API key set.\n") unless opt_chk_valid("fmi_api_key", 10);
238 a1d6e2d8789e Add new data from FMI service and change datafile format. Matti Hamalainen <ccr@tnsp.org> parents: 237 diff changeset	456 my @fmitems = ("temperature", "humidity", "windspeedms", "totalcloudcover");
158 db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	457
189 3af8ea19a3e2 Improved weather fetching. Matti Hamalainen <ccr@tnsp.org> parents: 186 diff changeset	458 my $uri = "http://data.fmi.fi/fmi-apikey/".opt_get("fmi_api_key").
3af8ea19a3e2 Improved weather fetching. Matti Hamalainen <ccr@tnsp.org> parents: 186 diff changeset	459 "/wfs?request=getFeature&storedquery_id=fmi::observations::weather::".
3af8ea19a3e2 Improved weather fetching. Matti Hamalainen <ccr@tnsp.org> parents: 186 diff changeset	460 "multipointcoverage".
3af8ea19a3e2 Improved weather fetching. Matti Hamalainen <ccr@tnsp.org> parents: 186 diff changeset	461 # "timevaluepair".
194 55ed3db51ee7 Rename a function. Matti Hamalainen <ccr@tnsp.org> parents: 193 diff changeset	462 "&starttime=".format_time_gmt(time() - 10*60)."&endtime=".format_time_gmt(time()).
238 a1d6e2d8789e Add new data from FMI service and change datafile format. Matti Hamalainen <ccr@tnsp.org> parents: 237 diff changeset	463 "&parameters=".join(",", @fmitems)."&maxlocations=100&bbox=19,59,32,75";
164 d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	464
324 19c64798de91 weather: Add some better optional debugging. Matti Hamalainen <ccr@tnsp.org> parents: 247 diff changeset	465 print STDERR "FMI URI: ".$uri."\n" if (opt_get_int("debug") > 0);
189 3af8ea19a3e2 Improved weather fetching. Matti Hamalainen <ccr@tnsp.org> parents: 186 diff changeset	466
3af8ea19a3e2 Improved weather fetching. Matti Hamalainen <ccr@tnsp.org> parents: 186 diff changeset	467 my $res = fetch_http($uri);
164 d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	468 if ($res->code >= 200 && $res->code <= 201)
d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	469 {
170 e4faa3746869 Remove useless parameters from XMLin(). Matti Hamalainen <ccr@tnsp.org> parents: 169 diff changeset	470 my $xml = XMLin($res->decoded_content);
169 c12e422a543a FMI data parsing kinda almost works. It is horrible, though. XML ... :Q Matti Hamalainen <ccr@tnsp.org> parents: 168 diff changeset	471 my $time_base = time();
339 6ea410acbe74 Cosmetics. Matti Hamalainen <ccr@tnsp.org> parents: 338 diff changeset	472
208 c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	473 if (defined($xml->{"wfs:member"}{"omso:GridSeriesObservation"}))
c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	474 {
c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	475 my $fdata = $xml->{"wfs:member"}{"omso:GridSeriesObservation"};
c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	476 my $fshit = $fdata->{"om:result"}{"gmlcov:MultiPointCoverage"};
378 b706e8ebecf8 fetch_weather: Possibly improve FMI XML mess parsing. Matti Hamalainen <ccr@tnsp.org> parents: 377 diff changeset	477
b706e8ebecf8 fetch_weather: Possibly improve FMI XML mess parsing. Matti Hamalainen <ccr@tnsp.org> parents: 377 diff changeset	478 my @position_lines = split(/\n/, $fshit->{"gml:domainSet"}{"gmlcov:SimpleMultiPoint"}{"gmlcov:positions"});
b706e8ebecf8 fetch_weather: Possibly improve FMI XML mess parsing. Matti Hamalainen <ccr@tnsp.org> parents: 377 diff changeset	479 my @data_lines = split(/\n/, $fshit->{"gml:rangeSet"}{"gml:DataBlock"}{"gml:doubleOrNilReasonTupleList"});
208 c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	480 my @farray = ();
169 c12e422a543a FMI data parsing kinda almost works. It is horrible, though. XML ... :Q Matti Hamalainen <ccr@tnsp.org> parents: 168 diff changeset	481
378 b706e8ebecf8 fetch_weather: Possibly improve FMI XML mess parsing. Matti Hamalainen <ccr@tnsp.org> parents: 377 diff changeset	482 if (scalar(@position_lines) == scalar(@data_lines))
169 c12e422a543a FMI data parsing kinda almost works. It is horrible, though. XML ... :Q Matti Hamalainen <ccr@tnsp.org> parents: 168 diff changeset	483 {
378 b706e8ebecf8 fetch_weather: Possibly improve FMI XML mess parsing. Matti Hamalainen <ccr@tnsp.org> parents: 377 diff changeset	484 for (my $nline = 0; $nline < scalar(@position_lines); $nline++)
208 c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	485 {
382 5573c1681da9 fetch_weather: Trim split input data lines in FMI XML. Matti Hamalainen <ccr@tnsp.org> parents: 381 diff changeset	486 my $dline = str_trim($data_lines[$nline]);
5573c1681da9 fetch_weather: Trim split input data lines in FMI XML. Matti Hamalainen <ccr@tnsp.org> parents: 381 diff changeset	487 my $pline = str_trim($position_lines[$nline]);
378 b706e8ebecf8 fetch_weather: Possibly improve FMI XML mess parsing. Matti Hamalainen <ccr@tnsp.org> parents: 377 diff changeset	488
b706e8ebecf8 fetch_weather: Possibly improve FMI XML mess parsing. Matti Hamalainen <ccr@tnsp.org> parents: 377 diff changeset	489 my @fmatches = ($dline =~ /\s([\+\-]?\d+\.\d\|NaN)\s*/ig);
b706e8ebecf8 fetch_weather: Possibly improve FMI XML mess parsing. Matti Hamalainen <ccr@tnsp.org> parents: 377 diff changeset	490 if (scalar(@fmatches) != scalar(@fmitems))
b706e8ebecf8 fetch_weather: Possibly improve FMI XML mess parsing. Matti Hamalainen <ccr@tnsp.org> parents: 377 diff changeset	491 {
b706e8ebecf8 fetch_weather: Possibly improve FMI XML mess parsing. Matti Hamalainen <ccr@tnsp.org> parents: 377 diff changeset	492 print STDERR "Not enough items in scalar line #".$nline." (".
b706e8ebecf8 fetch_weather: Possibly improve FMI XML mess parsing. Matti Hamalainen <ccr@tnsp.org> parents: 377 diff changeset	493 scalar(@fmatches). " vs ".scalar(@fmitems)."): ".$dline."\n";
b706e8ebecf8 fetch_weather: Possibly improve FMI XML mess parsing. Matti Hamalainen <ccr@tnsp.org> parents: 377 diff changeset	494 }
b706e8ebecf8 fetch_weather: Possibly improve FMI XML mess parsing. Matti Hamalainen <ccr@tnsp.org> parents: 377 diff changeset	495 else
b706e8ebecf8 fetch_weather: Possibly improve FMI XML mess parsing. Matti Hamalainen <ccr@tnsp.org> parents: 377 diff changeset	496 {
b706e8ebecf8 fetch_weather: Possibly improve FMI XML mess parsing. Matti Hamalainen <ccr@tnsp.org> parents: 377 diff changeset	497 my $vtmp = {};
b706e8ebecf8 fetch_weather: Possibly improve FMI XML mess parsing. Matti Hamalainen <ccr@tnsp.org> parents: 377 diff changeset	498 for (my $fni = 0; $fni < scalar(@fmitems); $fni++)
b706e8ebecf8 fetch_weather: Possibly improve FMI XML mess parsing. Matti Hamalainen <ccr@tnsp.org> parents: 377 diff changeset	499 {
b706e8ebecf8 fetch_weather: Possibly improve FMI XML mess parsing. Matti Hamalainen <ccr@tnsp.org> parents: 377 diff changeset	500 $$vtmp{$fmitems[$fni]} = $fmatches[$fni] if (lc($fmatches[$fni]) ne "nan");
b706e8ebecf8 fetch_weather: Possibly improve FMI XML mess parsing. Matti Hamalainen <ccr@tnsp.org> parents: 377 diff changeset	501 }
b706e8ebecf8 fetch_weather: Possibly improve FMI XML mess parsing. Matti Hamalainen <ccr@tnsp.org> parents: 377 diff changeset	502 if ($pline =~ /^\s([\+\-]?\d+\.\d)\s+([\+\-]?\d+\.\d)\s+(\d+)\s$/)
b706e8ebecf8 fetch_weather: Possibly improve FMI XML mess parsing. Matti Hamalainen <ccr@tnsp.org> parents: 377 diff changeset	503 {
b706e8ebecf8 fetch_weather: Possibly improve FMI XML mess parsing. Matti Hamalainen <ccr@tnsp.org> parents: 377 diff changeset	504 $$vtmp{"lat"} = $1;
b706e8ebecf8 fetch_weather: Possibly improve FMI XML mess parsing. Matti Hamalainen <ccr@tnsp.org> parents: 377 diff changeset	505 $$vtmp{"long"} = $2;
b706e8ebecf8 fetch_weather: Possibly improve FMI XML mess parsing. Matti Hamalainen <ccr@tnsp.org> parents: 377 diff changeset	506 $$vtmp{"time"} = $3;
b706e8ebecf8 fetch_weather: Possibly improve FMI XML mess parsing. Matti Hamalainen <ccr@tnsp.org> parents: 377 diff changeset	507 push(@farray, $vtmp);
b706e8ebecf8 fetch_weather: Possibly improve FMI XML mess parsing. Matti Hamalainen <ccr@tnsp.org> parents: 377 diff changeset	508 }
b706e8ebecf8 fetch_weather: Possibly improve FMI XML mess parsing. Matti Hamalainen <ccr@tnsp.org> parents: 377 diff changeset	509 else
b706e8ebecf8 fetch_weather: Possibly improve FMI XML mess parsing. Matti Hamalainen <ccr@tnsp.org> parents: 377 diff changeset	510 {
b706e8ebecf8 fetch_weather: Possibly improve FMI XML mess parsing. Matti Hamalainen <ccr@tnsp.org> parents: 377 diff changeset	511 print STDERR "Data mismatch #".$nline.": ".$pline."\n";
b706e8ebecf8 fetch_weather: Possibly improve FMI XML mess parsing. Matti Hamalainen <ccr@tnsp.org> parents: 377 diff changeset	512 }
b706e8ebecf8 fetch_weather: Possibly improve FMI XML mess parsing. Matti Hamalainen <ccr@tnsp.org> parents: 377 diff changeset	513 }
208 c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	514 }
169 c12e422a543a FMI data parsing kinda almost works. It is horrible, though. XML ... :Q Matti Hamalainen <ccr@tnsp.org> parents: 168 diff changeset	515 }
378 b706e8ebecf8 fetch_weather: Possibly improve FMI XML mess parsing. Matti Hamalainen <ccr@tnsp.org> parents: 377 diff changeset	516 else
169 c12e422a543a FMI data parsing kinda almost works. It is horrible, though. XML ... :Q Matti Hamalainen <ccr@tnsp.org> parents: 168 diff changeset	517 {
384 a375c276ec4d fetch_weather: Typofix. Matti Hamalainen <ccr@tnsp.org> parents: 383 diff changeset	518 print STDERR "Position and data line counts do not match.\n";
378 b706e8ebecf8 fetch_weather: Possibly improve FMI XML mess parsing. Matti Hamalainen <ccr@tnsp.org> parents: 377 diff changeset	519 goto skip_it;
189 3af8ea19a3e2 Improved weather fetching. Matti Hamalainen <ccr@tnsp.org> parents: 186 diff changeset	520 }
208 c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	521 # XXX Hashify the array into lat/long keys
339 6ea410acbe74 Cosmetics. Matti Hamalainen <ccr@tnsp.org> parents: 338 diff changeset	522
208 c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	523 # This is horrible :S
352 19b363586b40 fetch_weather: Fix debug prints. Matti Hamalainen <ccr@tnsp.org> parents: 349 diff changeset	524 my $nrecords = 0;
410 5c97006c89e6 fetch_weather: Cleanups. Matti Hamalainen <ccr@tnsp.org> parents: 407 diff changeset	525 foreach my $xnode (@{$fdata->{"om:featureOfInterest"}{"sams:SF_SpatialSamplingFeature"}{"sams:shape"}{"gml:MultiPoint"}{"gml:pointMember"}})
189 3af8ea19a3e2 Improved weather fetching. Matti Hamalainen <ccr@tnsp.org> parents: 186 diff changeset	526 {
228 fe420576f284 The kludges intensify. Matti Hamalainen <ccr@tnsp.org> parents: 213 diff changeset	527 my $floc = $xnode->{"gml:Point"};
208 c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	528 if ($floc->{"gml:pos"} =~ /^\s([\+\-]?\d+\.\d)\s+([\+\-]?\d+\.\d)\s$/)
189 3af8ea19a3e2 Improved weather fetching. Matti Hamalainen <ccr@tnsp.org> parents: 186 diff changeset	529 {
208 c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	530 my ($flat, $flong) = ($1, $2);
349 f7793ef7c925 fetch_weather: Improve debugging. Matti Hamalainen <ccr@tnsp.org> parents: 343 diff changeset	531
208 c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	532 # Should use a hash -
410 5c97006c89e6 fetch_weather: Cleanups. Matti Hamalainen <ccr@tnsp.org> parents: 407 diff changeset	533 foreach my $frec (@farray)
189 3af8ea19a3e2 Improved weather fetching. Matti Hamalainen <ccr@tnsp.org> parents: 186 diff changeset	534 {
410 5c97006c89e6 fetch_weather: Cleanups. Matti Hamalainen <ccr@tnsp.org> parents: 407 diff changeset	535 if ($frec->{"lat"} == $flat && $frec->{"long"} == $flong &&
383 1ba1191da15d fetch_weather: Check for empty station name. Matti Hamalainen <ccr@tnsp.org> parents: 382 diff changeset	536 $floc->{"gml:name"} ne "")
208 c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	537 {
349 f7793ef7c925 fetch_weather: Improve debugging. Matti Hamalainen <ccr@tnsp.org> parents: 343 diff changeset	538 $nrecords++;
208 c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	539 $weatherdata->{$floc->{"gml:name"}} =
c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	540 [
c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	541 1,
410 5c97006c89e6 fetch_weather: Cleanups. Matti Hamalainen <ccr@tnsp.org> parents: 407 diff changeset	542
414 d623652df6b5 fetch_weather / weather: Add weather measurement station location data (GPS WGS-84 lat/long/height) to weather cache. Not used in weather.tcl yet, tho. Matti Hamalainen <ccr@tnsp.org> parents: 413 diff changeset	543 $frec->{"lat"},
d623652df6b5 fetch_weather / weather: Add weather measurement station location data (GPS WGS-84 lat/long/height) to weather cache. Not used in weather.tcl yet, tho. Matti Hamalainen <ccr@tnsp.org> parents: 413 diff changeset	544 $frec->{"long"},
d623652df6b5 fetch_weather / weather: Add weather measurement station location data (GPS WGS-84 lat/long/height) to weather cache. Not used in weather.tcl yet, tho. Matti Hamalainen <ccr@tnsp.org> parents: 413 diff changeset	545 0,
d623652df6b5 fetch_weather / weather: Add weather measurement station location data (GPS WGS-84 lat/long/height) to weather cache. Not used in weather.tcl yet, tho. Matti Hamalainen <ccr@tnsp.org> parents: 413 diff changeset	546
410 5c97006c89e6 fetch_weather: Cleanups. Matti Hamalainen <ccr@tnsp.org> parents: 407 diff changeset	547 plonk_data($frec->{"time"}),
5c97006c89e6 fetch_weather: Cleanups. Matti Hamalainen <ccr@tnsp.org> parents: 407 diff changeset	548 plonk_data($frec->{"temperature"}),
238 a1d6e2d8789e Add new data from FMI service and change datafile format. Matti Hamalainen <ccr@tnsp.org> parents: 237 diff changeset	549
410 5c97006c89e6 fetch_weather: Cleanups. Matti Hamalainen <ccr@tnsp.org> parents: 407 diff changeset	550 plonk_data($frec->{"humidity"}),
5c97006c89e6 fetch_weather: Cleanups. Matti Hamalainen <ccr@tnsp.org> parents: 407 diff changeset	551 plonk_data($frec->{"windspeedms"}),
5c97006c89e6 fetch_weather: Cleanups. Matti Hamalainen <ccr@tnsp.org> parents: 407 diff changeset	552 translate_clouds(plonk_data($frec->{"totalcloudcover"})),
208 c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	553 ];
c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	554 }
189 3af8ea19a3e2 Improved weather fetching. Matti Hamalainen <ccr@tnsp.org> parents: 186 diff changeset	555 }
3af8ea19a3e2 Improved weather fetching. Matti Hamalainen <ccr@tnsp.org> parents: 186 diff changeset	556 }
169 c12e422a543a FMI data parsing kinda almost works. It is horrible, though. XML ... :Q Matti Hamalainen <ccr@tnsp.org> parents: 168 diff changeset	557 }
352 19b363586b40 fetch_weather: Fix debug prints. Matti Hamalainen <ccr@tnsp.org> parents: 349 diff changeset	558 print STDERR $nrecords." records from FMI.\n" if (opt_get_int("debug") > 0);
353 a52a983c0c14 Cosmetics. Matti Hamalainen <ccr@tnsp.org> parents: 352 diff changeset	559 }
a52a983c0c14 Cosmetics. Matti Hamalainen <ccr@tnsp.org> parents: 352 diff changeset	560 else
a52a983c0c14 Cosmetics. Matti Hamalainen <ccr@tnsp.org> parents: 352 diff changeset	561 {
208 c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	562 # defined
c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	563 print STDERR "Invalid XML received:\n";
c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	564 print STDERR $res->decoded_content."\n\n";
169 c12e422a543a FMI data parsing kinda almost works. It is horrible, though. XML ... :Q Matti Hamalainen <ccr@tnsp.org> parents: 168 diff changeset	565 }
353 a52a983c0c14 Cosmetics. Matti Hamalainen <ccr@tnsp.org> parents: 352 diff changeset	566 }
a52a983c0c14 Cosmetics. Matti Hamalainen <ccr@tnsp.org> parents: 352 diff changeset	567 else
a52a983c0c14 Cosmetics. Matti Hamalainen <ccr@tnsp.org> parents: 352 diff changeset	568 {
208 c6bbf69afc57 Add some error handling and debugging. Matti Hamalainen <ccr@tnsp.org> parents: 206 diff changeset	569 print STDERR "Error fetching FMI XML: ".$res->status_line."\n";
164 d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	570 }
158 db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	571 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	572
164 d5a0e4248f3e weather: Implement configuration file support for weather fetcher backend. Matti Hamalainen <ccr@tnsp.org> parents: 162 diff changeset	573
375 c303ddc0a683 fetch_weather: Don't die() in case of bad XML from FMI, just skip parsing Matti Hamalainen <ccr@tnsp.org> parents: 374 diff changeset	574 ### Skip here if the FMI shit fails due to broken data
c303ddc0a683 fetch_weather: Don't die() in case of bad XML from FMI, just skip parsing Matti Hamalainen <ccr@tnsp.org> parents: 374 diff changeset	575 skip_it:
c303ddc0a683 fetch_weather: Don't die() in case of bad XML from FMI, just skip parsing Matti Hamalainen <ccr@tnsp.org> parents: 374 diff changeset	576
c303ddc0a683 fetch_weather: Don't die() in case of bad XML from FMI, just skip parsing Matti Hamalainen <ccr@tnsp.org> parents: 374 diff changeset	577
178 9b8ec700ede4 Clean up the weather data parser backend a bit. Matti Hamalainen <ccr@tnsp.org> parents: 170 diff changeset	578 ###
394 315f31e4f964 fetch_weather: Make purge_threshold option entirely optional, e.g. not set or 0 will disable purging. Matti Hamalainen <ccr@tnsp.org> parents: 392 diff changeset	579 ### Purge too old entries
371 36869897f21d fetch_weather: Purge sufficiently old data. Matti Hamalainen <ccr@tnsp.org> parents: 370 diff changeset	580 ###
394 315f31e4f964 fetch_weather: Make purge_threshold option entirely optional, e.g. not set or 0 will disable purging. Matti Hamalainen <ccr@tnsp.org> parents: 392 diff changeset	581 if (opt_chk_valid("purge_threshold", 1))
315f31e4f964 fetch_weather: Make purge_threshold option entirely optional, e.g. not set or 0 will disable purging. Matti Hamalainen <ccr@tnsp.org> parents: 392 diff changeset	582 {
315f31e4f964 fetch_weather: Make purge_threshold option entirely optional, e.g. not set or 0 will disable purging. Matti Hamalainen <ccr@tnsp.org> parents: 392 diff changeset	583 my $purge = opt_get_int("purge_threshold");
315f31e4f964 fetch_weather: Make purge_threshold option entirely optional, e.g. not set or 0 will disable purging. Matti Hamalainen <ccr@tnsp.org> parents: 392 diff changeset	584 if ($purge > 0)
315f31e4f964 fetch_weather: Make purge_threshold option entirely optional, e.g. not set or 0 will disable purging. Matti Hamalainen <ccr@tnsp.org> parents: 392 diff changeset	585 {
315f31e4f964 fetch_weather: Make purge_threshold option entirely optional, e.g. not set or 0 will disable purging. Matti Hamalainen <ccr@tnsp.org> parents: 392 diff changeset	586 my $wqtime = time();
395 33b5e7adc30c fetch_weather: Make purge_threshold output some information when debugging. Matti Hamalainen <ccr@tnsp.org> parents: 394 diff changeset	587 my $nold = scalar(keys %$weatherdata);
373 bc7746ca622e fetch_weather: Implement purge_threshold setting. Matti Hamalainen <ccr@tnsp.org> parents: 371 diff changeset	588
394 315f31e4f964 fetch_weather: Make purge_threshold option entirely optional, e.g. not set or 0 will disable purging. Matti Hamalainen <ccr@tnsp.org> parents: 392 diff changeset	589 foreach my $key (keys %$weatherdata)
315f31e4f964 fetch_weather: Make purge_threshold option entirely optional, e.g. not set or 0 will disable purging. Matti Hamalainen <ccr@tnsp.org> parents: 392 diff changeset	590 {
414 d623652df6b5 fetch_weather / weather: Add weather measurement station location data (GPS WGS-84 lat/long/height) to weather cache. Not used in weather.tcl yet, tho. Matti Hamalainen <ccr@tnsp.org> parents: 413 diff changeset	591 if ($wqtime - $weatherdata->{$key}[4] > (60 * $purge))
394 315f31e4f964 fetch_weather: Make purge_threshold option entirely optional, e.g. not set or 0 will disable purging. Matti Hamalainen <ccr@tnsp.org> parents: 392 diff changeset	592 {
315f31e4f964 fetch_weather: Make purge_threshold option entirely optional, e.g. not set or 0 will disable purging. Matti Hamalainen <ccr@tnsp.org> parents: 392 diff changeset	593 delete $$weatherdata{$key};
315f31e4f964 fetch_weather: Make purge_threshold option entirely optional, e.g. not set or 0 will disable purging. Matti Hamalainen <ccr@tnsp.org> parents: 392 diff changeset	594 }
315f31e4f964 fetch_weather: Make purge_threshold option entirely optional, e.g. not set or 0 will disable purging. Matti Hamalainen <ccr@tnsp.org> parents: 392 diff changeset	595 }
395 33b5e7adc30c fetch_weather: Make purge_threshold output some information when debugging. Matti Hamalainen <ccr@tnsp.org> parents: 394 diff changeset	596
33b5e7adc30c fetch_weather: Make purge_threshold output some information when debugging. Matti Hamalainen <ccr@tnsp.org> parents: 394 diff changeset	597 my $nnew = scalar(keys %$weatherdata);
33b5e7adc30c fetch_weather: Make purge_threshold output some information when debugging. Matti Hamalainen <ccr@tnsp.org> parents: 394 diff changeset	598 print STDERR "Purged data older than ".$purge." minutes, ".$nold." -> ".$nnew." = ".($nold - $nnew)." removed.\n" if (opt_get_int("debug") > 0);
371 36869897f21d fetch_weather: Purge sufficiently old data. Matti Hamalainen <ccr@tnsp.org> parents: 370 diff changeset	599 }
36869897f21d fetch_weather: Purge sufficiently old data. Matti Hamalainen <ccr@tnsp.org> parents: 370 diff changeset	600 }
36869897f21d fetch_weather: Purge sufficiently old data. Matti Hamalainen <ccr@tnsp.org> parents: 370 diff changeset	601
36869897f21d fetch_weather: Purge sufficiently old data. Matti Hamalainen <ccr@tnsp.org> parents: 370 diff changeset	602
36869897f21d fetch_weather: Purge sufficiently old data. Matti Hamalainen <ccr@tnsp.org> parents: 370 diff changeset	603 ###
158 db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	604 ### Output
178 9b8ec700ede4 Clean up the weather data parser backend a bit. Matti Hamalainen <ccr@tnsp.org> parents: 170 diff changeset	605 ###
353 a52a983c0c14 Cosmetics. Matti Hamalainen <ccr@tnsp.org> parents: 352 diff changeset	606 if (opt_chk_valid("outfile", 1))
a52a983c0c14 Cosmetics. Matti Hamalainen <ccr@tnsp.org> parents: 352 diff changeset	607 {
438 a17b7fdd03c0 fetch_weather: Simplify and check for existence of output file before trying to read the old one. Matti Hamalainen <ccr@tnsp.org> parents: 417 diff changeset	608 my $filename = opt_get("outfile");
a17b7fdd03c0 fetch_weather: Simplify and check for existence of output file before trying to read the old one. Matti Hamalainen <ccr@tnsp.org> parents: 417 diff changeset	609 print STDERR "Dumping data to output file '".$filename."'\n" if (opt_get_int("debug") > 0);
368 ed1c7f2d81df fetch_weather: Close STDOUT filehandle before reopening it, to make sure we Matti Hamalainen <ccr@tnsp.org> parents: 367 diff changeset	610 close(STDOUT);
438 a17b7fdd03c0 fetch_weather: Simplify and check for existence of output file before trying to read the old one. Matti Hamalainen <ccr@tnsp.org> parents: 417 diff changeset	611 open(STDOUT, '>', $filename) or die("Could not open output file '".$filename."': $!\n");
158 db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	612 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	613
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	614 binmode STDOUT, ':encoding(utf-8)';
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	615
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	616 foreach my $key (sort { $a cmp $b } keys %$weatherdata)
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	617 {
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	618 print STDOUT $key."\|".join("\|", @{$weatherdata->{$key}})."\n";
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	619 }
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	620
db2b1f74d994 Initial import of weather data fetching backend, written in Perl. Matti Hamalainen <ccr@tnsp.org> parents: diff changeset	621 close(STDOUT);

Mercurial > hg > egg-tcls

annotate fetch_weather.pl @ 438:a17b7fdd03c0