Mercurial > hg > egg-tcls
annotate fetch_feeds.tcl @ 460:dbe249968591
fetch_feeds: Add support for SNI in TLS and make note about requiring tcl-tls 1.7.13+
author | Matti Hamalainen <ccr@tnsp.org> |
---|---|
date | Mon, 30 Oct 2017 15:46:50 +0200 |
parents | e5810c52d376 |
children | 057e399d86f6 |
rev | line source |
---|---|
0 | 1 #!/usr/bin/tclsh |
1 | 2 # |
3 # NOTICE! Change above path to correct tclsh binary path! | |
4 # | |
268
96310b1c88fa
feeds: Improve config resiliency.
Matti Hamalainen <ccr@tnsp.org>
parents:
265
diff
changeset
|
5 ############################################################################## |
0 | 6 # |
425
e5810c52d376
Bump some copyright years and versions.
Matti Hamalainen <ccr@tnsp.org>
parents:
424
diff
changeset
|
7 # FeedCheck fetcher v1.1 by Matti 'ccr' Hamalainen <ccr@tnsp.org> |
e5810c52d376
Bump some copyright years and versions.
Matti Hamalainen <ccr@tnsp.org>
parents:
424
diff
changeset
|
8 # (C) Copyright 2008-2017 Tecnic Software productions (TNSP) |
0 | 9 # |
10 # This script is freely distributable under GNU GPL (version 2) license. | |
11 # | |
460
dbe249968591
fetch_feeds: Add support for SNI in TLS and make note about requiring tcl-tls 1.7.13+
Matti Hamalainen <ccr@tnsp.org>
parents:
425
diff
changeset
|
12 # NOTICE! NOTICE! This script REQUIRES tcl-tls 1.7.13+ if you wish to |
dbe249968591
fetch_feeds: Add support for SNI in TLS and make note about requiring tcl-tls 1.7.13+
Matti Hamalainen <ccr@tnsp.org>
parents:
425
diff
changeset
|
13 # support SSL/TLS https for URL checking. And you probably do. |
dbe249968591
fetch_feeds: Add support for SNI in TLS and make note about requiring tcl-tls 1.7.13+
Matti Hamalainen <ccr@tnsp.org>
parents:
425
diff
changeset
|
14 # |
268
96310b1c88fa
feeds: Improve config resiliency.
Matti Hamalainen <ccr@tnsp.org>
parents:
265
diff
changeset
|
15 ############################################################################## |
0 | 16 |
265
908edc54005a
feeds: Move configuration to separate file.
Matti Hamalainen <ccr@tnsp.org>
parents:
159
diff
changeset
|
17 ### The configuration should be in config.feeds in same directory |
908edc54005a
feeds: Move configuration to separate file.
Matti Hamalainen <ccr@tnsp.org>
parents:
159
diff
changeset
|
18 ### as this script. Or change the line below to point where ever |
908edc54005a
feeds: Move configuration to separate file.
Matti Hamalainen <ccr@tnsp.org>
parents:
159
diff
changeset
|
19 ### you wish. See "config.feeds.example" for an example config file. |
908edc54005a
feeds: Move configuration to separate file.
Matti Hamalainen <ccr@tnsp.org>
parents:
159
diff
changeset
|
20 source [file dirname [info script]]/config.feeds |
0 | 21 |
422
880a07485275
Add utl_ctime() to utillib and use it elsewhere.
Matti Hamalainen <ccr@tnsp.org>
parents:
350
diff
changeset
|
22 ### Required utillib.tcl |
880a07485275
Add utl_ctime() to utillib and use it elsewhere.
Matti Hamalainen <ccr@tnsp.org>
parents:
350
diff
changeset
|
23 source [file dirname [info script]]/utillib.tcl |
880a07485275
Add utl_ctime() to utillib and use it elsewhere.
Matti Hamalainen <ccr@tnsp.org>
parents:
350
diff
changeset
|
24 |
0 | 25 |
268
96310b1c88fa
feeds: Improve config resiliency.
Matti Hamalainen <ccr@tnsp.org>
parents:
265
diff
changeset
|
26 ############################################################################## |
139
3305e142eecc
Change feed fetcher to use SQLite3 backend.
Matti Hamalainen <ccr@tnsp.org>
parents:
114
diff
changeset
|
27 |
423
44c9128097cd
feeds: Remember to require sqlite3 package.
Matti Hamalainen <ccr@tnsp.org>
parents:
422
diff
changeset
|
28 package require sqlite3 |
0 | 29 package require http |
271 | 30 |
265
908edc54005a
feeds: Move configuration to separate file.
Matti Hamalainen <ccr@tnsp.org>
parents:
159
diff
changeset
|
31 if {[info exists http_user_agent] && $http_user_agent != ""} { |
296 | 32 ::http::config -urlencoding utf8 -useragent $http_user_agent |
265
908edc54005a
feeds: Move configuration to separate file.
Matti Hamalainen <ccr@tnsp.org>
parents:
159
diff
changeset
|
33 } else { |
322 | 34 ::http::config -urlencoding utf8 -useragent "Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Firefox/38.0" |
265
908edc54005a
feeds: Move configuration to separate file.
Matti Hamalainen <ccr@tnsp.org>
parents:
159
diff
changeset
|
35 } |
271 | 36 |
268
96310b1c88fa
feeds: Improve config resiliency.
Matti Hamalainen <ccr@tnsp.org>
parents:
265
diff
changeset
|
37 if {[info exists http_use_proxy] && $http_use_proxy != 0} { |
63 | 38 ::http::config -proxyhost $http_proxy_host -proxyport $http_proxy_port |
0 | 39 } |
40 | |
268
96310b1c88fa
feeds: Improve config resiliency.
Matti Hamalainen <ccr@tnsp.org>
parents:
265
diff
changeset
|
41 if {[info exists http_tls_support] && $http_tls_support != 0} { |
265
908edc54005a
feeds: Move configuration to separate file.
Matti Hamalainen <ccr@tnsp.org>
parents:
159
diff
changeset
|
42 package require tls |
460
dbe249968591
fetch_feeds: Add support for SNI in TLS and make note about requiring tcl-tls 1.7.13+
Matti Hamalainen <ccr@tnsp.org>
parents:
425
diff
changeset
|
43 ::http::register https 443 [list ::tls::socket -request true -require true -ssl2 false -ssl3 false -tls1 true -tls1.1 true -tls1.2 true -cadir $http_tls_cadir -autoservername true] |
265
908edc54005a
feeds: Move configuration to separate file.
Matti Hamalainen <ccr@tnsp.org>
parents:
159
diff
changeset
|
44 } |
908edc54005a
feeds: Move configuration to separate file.
Matti Hamalainen <ccr@tnsp.org>
parents:
159
diff
changeset
|
45 |
0 | 46 |
268
96310b1c88fa
feeds: Improve config resiliency.
Matti Hamalainen <ccr@tnsp.org>
parents:
265
diff
changeset
|
47 ############################################################################## |
96310b1c88fa
feeds: Improve config resiliency.
Matti Hamalainen <ccr@tnsp.org>
parents:
265
diff
changeset
|
48 |
321
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
49 proc fetch_dorequest { urlStr urlStatus urlSCode urlCode urlData urlMeta } { |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
50 upvar 1 $urlStatus ustatus |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
51 upvar 1 $urlSCode uscode |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
52 upvar 1 $urlCode ucode |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
53 upvar 1 $urlData udata |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
54 upvar 1 $urlMeta umeta |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
55 |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
56 if {[catch {set utoken [::http::geturl $urlStr -timeout 6000 -binary 1 -headers {Accept-Encoding identity}]} uerrmsg]} { |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
57 puts "HTTP request failed: $uerrmsg" |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
58 return 0 |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
59 } |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
60 |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
61 set ustatus [::http::status $utoken] |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
62 if {$ustatus == "timeout"} { |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
63 puts "HTTP request timed out ($urlStr)" |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
64 return 0 |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
65 } |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
66 |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
67 if {$ustatus != "ok"} { |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
68 puts "Error in HTTP transaction: [::http::error $utoken] ($urlStr)" |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
69 return 0 |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
70 } |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
71 |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
72 set ustatus [::http::status $utoken] |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
73 set uscode [::http::code $utoken] |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
74 set ucode [::http::ncode $utoken] |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
75 set udata [::http::data $utoken] |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
76 array set umeta [::http::meta $utoken] |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
77 ::http::cleanup $utoken |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
78 |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
79 return 1 |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
80 } |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
81 |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
82 |
139
3305e142eecc
Change feed fetcher to use SQLite3 backend.
Matti Hamalainen <ccr@tnsp.org>
parents:
114
diff
changeset
|
83 proc add_entry {uname uprefix uurl utitle} { |
142 | 84 global currclock feeds_db nitems |
292
9f90d6918626
feeds: Also use the html entity conversion from utillib here.
Matti Hamalainen <ccr@tnsp.org>
parents:
271
diff
changeset
|
85 set utmp [utl_convert_html_ent $uurl] |
147 | 86 if {[string match "http://*" $utmp] || [string match "https://*" $utmp]} { |
87 set utest "$utmp" | |
88 } else { | |
321
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
89 if {[string range $uprefix end end] != "/" && [string range $utmp 0 0] != "/"} { |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
90 set utest "$uprefix/$utmp" |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
91 } else { |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
92 set utest "$uprefix$utmp" |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
93 } |
147 | 94 } |
139
3305e142eecc
Change feed fetcher to use SQLite3 backend.
Matti Hamalainen <ccr@tnsp.org>
parents:
114
diff
changeset
|
95 |
296 | 96 set usql "SELECT title FROM feeds WHERE url='[utl_escape $utest]' AND feed='[utl_escape $uname]'" |
140
b0648e05c855
Change some variable names, etc.
Matti Hamalainen <ccr@tnsp.org>
parents:
139
diff
changeset
|
97 if {![feeds_db exists $usql]} { |
321
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
98 # puts "NEW: $utest : $utitle" |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
99 set usql "INSERT INTO feeds (feed,utime,url,title) VALUES ('[utl_escape $uname]', $currclock, '[utl_escape $utest]', '[utl_escape [utl_convert_html_ent $utitle]]')" |
142 | 100 incr nitems |
140
b0648e05c855
Change some variable names, etc.
Matti Hamalainen <ccr@tnsp.org>
parents:
139
diff
changeset
|
101 if {[catch {feeds_db eval $usql} uerrmsg]} { |
139
3305e142eecc
Change feed fetcher to use SQLite3 backend.
Matti Hamalainen <ccr@tnsp.org>
parents:
114
diff
changeset
|
102 puts "\nError: $uerrmsg on:\n$usql" |
3305e142eecc
Change feed fetcher to use SQLite3 backend.
Matti Hamalainen <ccr@tnsp.org>
parents:
114
diff
changeset
|
103 exit 15 |
3305e142eecc
Change feed fetcher to use SQLite3 backend.
Matti Hamalainen <ccr@tnsp.org>
parents:
114
diff
changeset
|
104 } |
63 | 105 } |
0 | 106 } |
107 | |
108 | |
109 proc add_rss_feed {datauri dataname dataprefix} { | |
321
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
110 if {[catch {set utoken [::http::geturl $datauri -binary 1 -timeout 6000 -headers {Accept-Encoding identity}]} uerrmsg]} { |
63 | 111 puts "Error getting $datauri: $uerrmsg" |
112 return 1 | |
113 } | |
114 set upage [::http::data $utoken] | |
115 ::http::cleanup $utoken | |
424
825cac46b1cb
Cosmetic / stray trailing whitespace cleanup.
Matti Hamalainen <ccr@tnsp.org>
parents:
423
diff
changeset
|
116 |
63 | 117 set umatches [regexp -all -nocase -inline -- "<item>.\*\?<title><..CDATA.(.\*\?)\\\]\\\]></title>.\*\?<link>(http.\*\?)</link>.\*\?</item>" $upage] |
118 set nmatches [llength $umatches] | |
119 for {set n 0} {$n < $nmatches} {incr n 3} { | |
120 add_entry $dataname $dataprefix [lindex $umatches [expr $n+2]] [lindex $umatches [expr $n+1]] | |
121 } | |
424
825cac46b1cb
Cosmetic / stray trailing whitespace cleanup.
Matti Hamalainen <ccr@tnsp.org>
parents:
423
diff
changeset
|
122 |
63 | 123 if {$nmatches == 0} { |
124 set umatches [regexp -all -nocase -inline -- "<item>.\*\?<title>(.\*\?)</title>.\*\?<link>(http.\*\?)</link>.\*\?</item>" $upage] | |
125 set nmatches [llength $umatches] | |
126 for {set n 0} {$n < $nmatches} {incr n 3} { | |
127 add_entry $dataname $dataprefix [lindex $umatches [expr $n+2]] [lindex $umatches [expr $n+1]] | |
128 } | |
129 } | |
0 | 130 |
63 | 131 if {$nmatches == 0} { |
132 set umatches [regexp -all -nocase -inline -- "<item \[^>\]*>.\*\?<title>(.\*\?)</title>.\*\?<link>(http.\*\?)</link>.\*\?</item>" $upage] | |
133 set nmatches [llength $umatches] | |
134 for {set n 0} {$n < $nmatches} {incr n 3} { | |
135 add_entry $dataname $dataprefix [lindex $umatches [expr $n+2]] [lindex $umatches [expr $n+1]] | |
136 } | |
137 } | |
143 | 138 |
63 | 139 return 0 |
0 | 140 } |
141 | |
142 | |
143 ############################################################################## | |
69
df3230f8aa46
Translate some comments to english and cosmetic fixes.
Matti Hamalainen <ccr@tnsp.org>
parents:
63
diff
changeset
|
144 ### Fetch and parse Halla-aho's blog page data |
321
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
145 proc fetch_halla_aho { } { |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
146 set datauri "http://www.halla-aho.com/scripta/"; |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
147 set dataname "Mestari" |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
148 if {![fetch_dorequest $datauri ustatus uscode ucode upage umeta]} { |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
149 return 0 |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
150 } |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
151 |
63 | 152 set umatches [regexp -all -nocase -inline -- "<a href=\"(\[^\"\]+\.html)\"><b>(\[^<\]+)</b>" $upage] |
153 set nmatches [llength $umatches] | |
154 for {set n 0} {$n < $nmatches} {incr n 3} { | |
155 add_entry $dataname $datauri [lindex $umatches [expr $n+1]] [lindex $umatches [expr $n+2]] | |
156 } | |
0 | 157 |
63 | 158 set umatches [regexp -all -nocase -inline -- "<a href=\"(\[^\"\]+\.html)\">(\[^<\]\[^b\]\[^<\]+)</a>" $upage] |
159 set nmatches [llength $umatches] | |
160 for {set n 0} {$n < $nmatches} {incr n 3} { | |
161 add_entry $dataname $datauri [lindex $umatches [expr $n+1]] [lindex $umatches [expr $n+2]] | |
162 } | |
0 | 163 } |
164 | |
165 | |
166 ### The Adventurers | |
321
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
167 proc fetch_adventurers { } { |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
168 set datauri "http://www.peldor.com/chapters/index_sidebar.html"; |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
169 set dataname "The Adventurers" |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
170 if {![fetch_dorequest $datauri ustatus uscode ucode upage umeta]} { |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
171 return 0 |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
172 } |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
173 |
63 | 174 set umatches [regexp -all -nocase -inline -- "<a href=\"(\[^\"\]+)\">(\[^<\]+)</a>" $upage] |
175 set nmatches [llength $umatches] | |
176 for {set n 0} {$n < $nmatches} {incr n 3} { | |
177 add_entry $dataname "http://www.peldor.com/" [lindex $umatches [expr $n+1]] [lindex $umatches [expr $n+2]] | |
178 } | |
0 | 179 } |
180 | |
181 | |
182 ### Order of the Stick | |
321
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
183 proc fetch_oots { } { |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
184 set datauri "http://www.giantitp.com/comics/oots.html"; |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
185 set dataname "OOTS" |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
186 if {![fetch_dorequest $datauri ustatus uscode ucode upage umeta]} { |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
187 return 0 |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
188 } |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
189 |
63 | 190 set umatches [regexp -all -nocase -inline -- "<a href=\"(/comics/oots\[0-9\]+\.html)\">(\[^<\]+)</a>" $upage] |
191 set nmatches [llength $umatches] | |
192 for {set n 0} {$n < $nmatches} {incr n 3} { | |
193 add_entry $dataname "http://www.giantitp.com" [lindex $umatches [expr $n+1]] [lindex $umatches [expr $n+2]] | |
194 } | |
0 | 195 } |
196 | |
197 | |
350
51c08336d7b1
feeds: Add support for Poliisi.fi information reports.
Matti Hamalainen <ccr@tnsp.org>
parents:
323
diff
changeset
|
198 ### Poliisi tiedotteet |
51c08336d7b1
feeds: Add support for Poliisi.fi information reports.
Matti Hamalainen <ccr@tnsp.org>
parents:
323
diff
changeset
|
199 proc fetch_poliisi { datauri dataname dataprefix } { |
51c08336d7b1
feeds: Add support for Poliisi.fi information reports.
Matti Hamalainen <ccr@tnsp.org>
parents:
323
diff
changeset
|
200 if {![fetch_dorequest $datauri ustatus uscode ucode upage umeta]} { |
51c08336d7b1
feeds: Add support for Poliisi.fi information reports.
Matti Hamalainen <ccr@tnsp.org>
parents:
323
diff
changeset
|
201 return 0 |
51c08336d7b1
feeds: Add support for Poliisi.fi information reports.
Matti Hamalainen <ccr@tnsp.org>
parents:
323
diff
changeset
|
202 } |
51c08336d7b1
feeds: Add support for Poliisi.fi information reports.
Matti Hamalainen <ccr@tnsp.org>
parents:
323
diff
changeset
|
203 |
51c08336d7b1
feeds: Add support for Poliisi.fi information reports.
Matti Hamalainen <ccr@tnsp.org>
parents:
323
diff
changeset
|
204 set umatches [regexp -all -nocase -inline -- "<div class=\"channelitem\"><div class=\"date\">(.*?)</div><a class=\"article\" href=\"(\[^\"\]+)\">(\[^<\]+)</a>" $upage] |
51c08336d7b1
feeds: Add support for Poliisi.fi information reports.
Matti Hamalainen <ccr@tnsp.org>
parents:
323
diff
changeset
|
205 set nmatches [llength $umatches] |
51c08336d7b1
feeds: Add support for Poliisi.fi information reports.
Matti Hamalainen <ccr@tnsp.org>
parents:
323
diff
changeset
|
206 for {set n 0} {$n < $nmatches} {incr n 4} { |
51c08336d7b1
feeds: Add support for Poliisi.fi information reports.
Matti Hamalainen <ccr@tnsp.org>
parents:
323
diff
changeset
|
207 set stmp [string trim [lindex $umatches [expr $n+3]]] |
51c08336d7b1
feeds: Add support for Poliisi.fi information reports.
Matti Hamalainen <ccr@tnsp.org>
parents:
323
diff
changeset
|
208 add_entry $dataname $dataprefix [lindex $umatches [expr $n+2]] "[lindex $umatches [expr $n+1]]: $stmp" |
51c08336d7b1
feeds: Add support for Poliisi.fi information reports.
Matti Hamalainen <ccr@tnsp.org>
parents:
323
diff
changeset
|
209 } |
51c08336d7b1
feeds: Add support for Poliisi.fi information reports.
Matti Hamalainen <ccr@tnsp.org>
parents:
323
diff
changeset
|
210 } |
51c08336d7b1
feeds: Add support for Poliisi.fi information reports.
Matti Hamalainen <ccr@tnsp.org>
parents:
323
diff
changeset
|
211 |
51c08336d7b1
feeds: Add support for Poliisi.fi information reports.
Matti Hamalainen <ccr@tnsp.org>
parents:
323
diff
changeset
|
212 |
51c08336d7b1
feeds: Add support for Poliisi.fi information reports.
Matti Hamalainen <ccr@tnsp.org>
parents:
323
diff
changeset
|
213 |
51c08336d7b1
feeds: Add support for Poliisi.fi information reports.
Matti Hamalainen <ccr@tnsp.org>
parents:
323
diff
changeset
|
214 |
321
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
215 ### Open database, etc |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
216 set nitems 0 |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
217 set currclock [clock seconds] |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
218 global feeds_db |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
219 if {[catch {sqlite3 feeds_db $feeds_dbfile} uerrmsg]} { |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
220 puts "Could not open SQLite3 database '$feeds_dbfile': $uerrmsg." |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
221 exit 2 |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
222 } |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
223 |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
224 |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
225 ### Fetch the feeds |
350
51c08336d7b1
feeds: Add support for Poliisi.fi information reports.
Matti Hamalainen <ccr@tnsp.org>
parents:
323
diff
changeset
|
226 fetch_poliisi "http://www.poliisi.fi/oulu/tiedotteet/1/0?all1/0" "Poliisi/Oulu" "http://www.poliisi.fi" |
51c08336d7b1
feeds: Add support for Poliisi.fi information reports.
Matti Hamalainen <ccr@tnsp.org>
parents:
323
diff
changeset
|
227 |
321
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
228 fetch_halla_aho |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
229 |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
230 fetch_adventurers |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
231 |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
232 fetch_oots |
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
233 |
143 | 234 #add_rss_feed "http://www.kaleva.fi/rss/145.xml" "Kaleva/Tiede" "" |
0 | 235 |
236 add_rss_feed "http://www.effi.org/xml/uutiset.rss" "EFFI" "" | |
237 | |
143 | 238 add_rss_feed "http://static.mtv3.fi/rss/uutiset_rikos.rss" "MTV3/Rikos" "" |
0 | 239 |
321
d8b957796121
feeds: Refactor the feeds fetching.
Matti Hamalainen <ccr@tnsp.org>
parents:
296
diff
changeset
|
240 #add_rss_feed "http://www.blastwave-comic.com/rss/blastwave.xml" "Blastwave" "" |
0 | 241 |
242 #add_rss_feed "http://lehti.samizdat.info/feed/" "Lehti" "" | |
243 | |
139
3305e142eecc
Change feed fetcher to use SQLite3 backend.
Matti Hamalainen <ccr@tnsp.org>
parents:
114
diff
changeset
|
244 |
3305e142eecc
Change feed fetcher to use SQLite3 backend.
Matti Hamalainen <ccr@tnsp.org>
parents:
114
diff
changeset
|
245 ### Close database |
140
b0648e05c855
Change some variable names, etc.
Matti Hamalainen <ccr@tnsp.org>
parents:
139
diff
changeset
|
246 feeds_db close |
142 | 247 |
248 puts "$nitems new items." |