Mercurial > hg > egg-tcls
annotate urllog.tcl @ 306:9858b93387a2
urllog: 100L.
| author | Matti Hamalainen <ccr@tnsp.org> |
|---|---|
| date | Tue, 27 Jan 2015 08:44:15 +0200 |
| parents | f1589fe20732 |
| children | adc519c72f53 |
| rev | line source |
|---|---|
| 0 | 1 ########################################################################## |
| 2 # | |
| 253 | 3 # URLLog v2.4.0 by Matti 'ccr' Hamalainen <ccr@tnsp.org> |
| 250 | 4 # (C) Copyright 2000-2015 Tecnic Software productions (TNSP) |
| 0 | 5 # |
|
113
077c7383f36f
urllog: Add line about the script's license.
Matti Hamalainen <ccr@tnsp.org>
parents:
112
diff
changeset
|
6 # This script is freely distributable under GNU GPL (version 2) license. |
|
077c7383f36f
urllog: Add line about the script's license.
Matti Hamalainen <ccr@tnsp.org>
parents:
112
diff
changeset
|
7 # |
| 0 | 8 ########################################################################## |
| 9 # | |
|
50
f69363fc1f61
Update some comments and add a bit of documentation.
Matti Hamalainen <ccr@tnsp.org>
parents:
49
diff
changeset
|
10 # URL-logger script for EggDrop IRC robot, utilizing SQLite3 database |
|
81
17e542b7985a
urllog, quotedb: Improve documentation.
Matti Hamalainen <ccr@tnsp.org>
parents:
73
diff
changeset
|
11 # This script requires SQLite TCL extension. Under Debian, you need: |
|
17e542b7985a
urllog, quotedb: Improve documentation.
Matti Hamalainen <ccr@tnsp.org>
parents:
73
diff
changeset
|
12 # tcl8.5 libsqlite3-tcl (and eggdrop eggdrop-data, of course) |
|
50
f69363fc1f61
Update some comments and add a bit of documentation.
Matti Hamalainen <ccr@tnsp.org>
parents:
49
diff
changeset
|
13 # |
|
81
17e542b7985a
urllog, quotedb: Improve documentation.
Matti Hamalainen <ccr@tnsp.org>
parents:
73
diff
changeset
|
14 # NOTICE! If you are upgrading to URLLog v2.0+ from any 1.x version, you |
|
50
f69363fc1f61
Update some comments and add a bit of documentation.
Matti Hamalainen <ccr@tnsp.org>
parents:
49
diff
changeset
|
15 # may want to run a conversion script against your URL-database file, |
|
f69363fc1f61
Update some comments and add a bit of documentation.
Matti Hamalainen <ccr@tnsp.org>
parents:
49
diff
changeset
|
16 # if you wish to preserve the old data. |
| 0 | 17 # |
|
50
f69363fc1f61
Update some comments and add a bit of documentation.
Matti Hamalainen <ccr@tnsp.org>
parents:
49
diff
changeset
|
18 # See convert_urllog_db.tcl for more information. |
|
f69363fc1f61
Update some comments and add a bit of documentation.
Matti Hamalainen <ccr@tnsp.org>
parents:
49
diff
changeset
|
19 # |
|
81
17e542b7985a
urllog, quotedb: Improve documentation.
Matti Hamalainen <ccr@tnsp.org>
parents:
73
diff
changeset
|
20 # If you are doing a fresh install, you will need to create the |
|
50
f69363fc1f61
Update some comments and add a bit of documentation.
Matti Hamalainen <ccr@tnsp.org>
parents:
49
diff
changeset
|
21 # initial SQLite3 database with the required table schemas. You |
|
f69363fc1f61
Update some comments and add a bit of documentation.
Matti Hamalainen <ccr@tnsp.org>
parents:
49
diff
changeset
|
22 # can do that by running: create_urllog_db.tcl |
| 0 | 23 # |
| 24 ########################################################################## | |
|
13
e06d41fb69d5
Begin work on converting urllog.tcl to use an SQLite3 database instead of flat file.
Matti Hamalainen <ccr@tnsp.org>
parents:
8
diff
changeset
|
25 |
|
263
f01d60175c44
urllog: Move configuration to external file.
Matti Hamalainen <ccr@tnsp.org>
parents:
260
diff
changeset
|
26 ### The configuration should be in config.urllog in same directory |
|
f01d60175c44
urllog: Move configuration to external file.
Matti Hamalainen <ccr@tnsp.org>
parents:
260
diff
changeset
|
27 ### as this script. Or change the line below to point where ever |
|
f01d60175c44
urllog: Move configuration to external file.
Matti Hamalainen <ccr@tnsp.org>
parents:
260
diff
changeset
|
28 ### you wish. See "config.urllog.example" for an example config file. |
|
f01d60175c44
urllog: Move configuration to external file.
Matti Hamalainen <ccr@tnsp.org>
parents:
260
diff
changeset
|
29 source [file dirname [info script]]/config.urllog |
| 0 | 30 |
|
291
54d34d086b47
urllog: Use the utility lib for entity conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
289
diff
changeset
|
31 ### Required utillib.tcl |
|
54d34d086b47
urllog: Use the utility lib for entity conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
289
diff
changeset
|
32 source [file dirname [info script]]/utillib.tcl |
|
54d34d086b47
urllog: Use the utility lib for entity conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
289
diff
changeset
|
33 |
| 0 | 34 |
| 35 ########################################################################## | |
| 36 # No need to look below this line | |
| 37 ########################################################################## | |
| 38 set urllog_name "URLLog" | |
| 253 | 39 set urllog_version "2.4.0" |
| 0 | 40 |
|
300
2a9ee3f68225
urllog: Make TLD check configurable.
Matti Hamalainen <ccr@tnsp.org>
parents:
299
diff
changeset
|
41 set urllog_tld_list [split $urllog_tld_list ","] |
| 0 | 42 set urllog_httprep [split "\@|%40|{|%7B|}|%7D|\[|%5B|\]|%5D" "|"] |
| 43 | |
|
102
5425dc418505
urllog: Entity data is now in UTF-8, but TCL source files are interpreted with current system locale, which may not be UTF-8. We must therefore "convert" the entity mapping string to UTF-8 to be certain of TCL's interpretation of its encoding.
Matti Hamalainen <ccr@tnsp.org>
parents:
101
diff
changeset
|
44 |
|
13
e06d41fb69d5
Begin work on converting urllog.tcl to use an SQLite3 database instead of flat file.
Matti Hamalainen <ccr@tnsp.org>
parents:
8
diff
changeset
|
45 ### Require packages |
|
e06d41fb69d5
Begin work on converting urllog.tcl to use an SQLite3 database instead of flat file.
Matti Hamalainen <ccr@tnsp.org>
parents:
8
diff
changeset
|
46 package require sqlite3 |
| 0 | 47 package require http |
|
7
50b52294e93e
urllog: Strip ‏ entities from titles; Some work on SSL/https support.
Matti Hamalainen <ccr@tnsp.org>
parents:
4
diff
changeset
|
48 |
| 0 | 49 ### Binding initializations |
|
219
4e09bcc48851
urllog: Add settings for specifying channels where URL logging is active, and where !urlfind functionality works (separately, if so desired.)
Matti Hamalainen <ccr@tnsp.org>
parents:
218
diff
changeset
|
50 bind pub - !urlfind urllog_pub_urlfind |
|
4e09bcc48851
urllog: Add settings for specifying channels where URL logging is active, and where !urlfind functionality works (separately, if so desired.)
Matti Hamalainen <ccr@tnsp.org>
parents:
218
diff
changeset
|
51 bind msg - !urlfind urllog_msg_urlfind |
| 249 | 52 bind pubm - *.* urllog_check_line |
| 53 bind topc - *.* urllog_check_line | |
| 0 | 54 |
| 55 | |
| 56 ### Initialization messages | |
| 250 | 57 set urllog_message "$urllog_name v$urllog_version (C) 2000-2015 ccr/TNSP" |
| 0 | 58 putlog "$urllog_message" |
| 59 | |
|
289
5067843cee3d
urllog: Move some initialization messages around.
Matti Hamalainen <ccr@tnsp.org>
parents:
269
diff
changeset
|
60 ### Miscellaneous init messages |
|
5067843cee3d
urllog: Move some initialization messages around.
Matti Hamalainen <ccr@tnsp.org>
parents:
269
diff
changeset
|
61 if {$urllog_extra_checks != 0} { |
|
5067843cee3d
urllog: Move some initialization messages around.
Matti Hamalainen <ccr@tnsp.org>
parents:
269
diff
changeset
|
62 putlog " (Additional URL validity checks enabled)" |
|
5067843cee3d
urllog: Move some initialization messages around.
Matti Hamalainen <ccr@tnsp.org>
parents:
269
diff
changeset
|
63 } |
|
5067843cee3d
urllog: Move some initialization messages around.
Matti Hamalainen <ccr@tnsp.org>
parents:
269
diff
changeset
|
64 |
|
300
2a9ee3f68225
urllog: Make TLD check configurable.
Matti Hamalainen <ccr@tnsp.org>
parents:
299
diff
changeset
|
65 if {$urllog_check_tld != 0} { |
|
2a9ee3f68225
urllog: Make TLD check configurable.
Matti Hamalainen <ccr@tnsp.org>
parents:
299
diff
changeset
|
66 putlog " (Check TLD)" |
|
2a9ee3f68225
urllog: Make TLD check configurable.
Matti Hamalainen <ccr@tnsp.org>
parents:
299
diff
changeset
|
67 } |
|
2a9ee3f68225
urllog: Make TLD check configurable.
Matti Hamalainen <ccr@tnsp.org>
parents:
299
diff
changeset
|
68 |
|
289
5067843cee3d
urllog: Move some initialization messages around.
Matti Hamalainen <ccr@tnsp.org>
parents:
269
diff
changeset
|
69 if {$urllog_verbose != 0} { |
|
5067843cee3d
urllog: Move some initialization messages around.
Matti Hamalainen <ccr@tnsp.org>
parents:
269
diff
changeset
|
70 putlog " (Verbose mode enabled)" |
|
5067843cee3d
urllog: Move some initialization messages around.
Matti Hamalainen <ccr@tnsp.org>
parents:
269
diff
changeset
|
71 } |
|
5067843cee3d
urllog: Move some initialization messages around.
Matti Hamalainen <ccr@tnsp.org>
parents:
269
diff
changeset
|
72 |
|
13
e06d41fb69d5
Begin work on converting urllog.tcl to use an SQLite3 database instead of flat file.
Matti Hamalainen <ccr@tnsp.org>
parents:
8
diff
changeset
|
73 ### HTTP module initialization |
|
269
d62280f2a9c7
urllog: Make user agent string configurable.
Matti Hamalainen <ccr@tnsp.org>
parents:
267
diff
changeset
|
74 if {[info exists http_user_agent] && $http_user_agent != ""} { |
|
d62280f2a9c7
urllog: Make user agent string configurable.
Matti Hamalainen <ccr@tnsp.org>
parents:
267
diff
changeset
|
75 ::http::config -useragent $http_user_agent |
|
d62280f2a9c7
urllog: Make user agent string configurable.
Matti Hamalainen <ccr@tnsp.org>
parents:
267
diff
changeset
|
76 } else { |
|
d62280f2a9c7
urllog: Make user agent string configurable.
Matti Hamalainen <ccr@tnsp.org>
parents:
267
diff
changeset
|
77 ::http::config -useragent "$urllog_name/$urllog_version" |
|
d62280f2a9c7
urllog: Make user agent string configurable.
Matti Hamalainen <ccr@tnsp.org>
parents:
267
diff
changeset
|
78 } |
|
d62280f2a9c7
urllog: Make user agent string configurable.
Matti Hamalainen <ccr@tnsp.org>
parents:
267
diff
changeset
|
79 |
|
267
da239a953e24
urllog: Change some setting names, etc.
Matti Hamalainen <ccr@tnsp.org>
parents:
264
diff
changeset
|
80 if {[info exists http_use_proxy] && $http_use_proxy != 0} { |
| 28 | 81 ::http::config -proxyhost $http_proxy_host -proxyport $http_proxy_port |
|
267
da239a953e24
urllog: Change some setting names, etc.
Matti Hamalainen <ccr@tnsp.org>
parents:
264
diff
changeset
|
82 putlog " (Using proxy $http_proxy_host:$http_proxy_port)" |
|
13
e06d41fb69d5
Begin work on converting urllog.tcl to use an SQLite3 database instead of flat file.
Matti Hamalainen <ccr@tnsp.org>
parents:
8
diff
changeset
|
83 } |
|
e06d41fb69d5
Begin work on converting urllog.tcl to use an SQLite3 database instead of flat file.
Matti Hamalainen <ccr@tnsp.org>
parents:
8
diff
changeset
|
84 |
|
267
da239a953e24
urllog: Change some setting names, etc.
Matti Hamalainen <ccr@tnsp.org>
parents:
264
diff
changeset
|
85 if {[info exists http_tls_support] && $http_tls_support != 0} { |
| 28 | 86 package require tls |
|
235
059660980388
urllog: Enable TLS, fixes annoying issues where https fails.
Matti Hamalainen <ccr@tnsp.org>
parents:
230
diff
changeset
|
87 ::http::register https 443 [list ::tls::socket -request 1 -require 1 -tls1 1 -cadir $http_tls_cadir] |
|
267
da239a953e24
urllog: Change some setting names, etc.
Matti Hamalainen <ccr@tnsp.org>
parents:
264
diff
changeset
|
88 putlog " (TLS/SSL support enabled)" |
|
13
e06d41fb69d5
Begin work on converting urllog.tcl to use an SQLite3 database instead of flat file.
Matti Hamalainen <ccr@tnsp.org>
parents:
8
diff
changeset
|
89 } |
|
e06d41fb69d5
Begin work on converting urllog.tcl to use an SQLite3 database instead of flat file.
Matti Hamalainen <ccr@tnsp.org>
parents:
8
diff
changeset
|
90 |
|
e06d41fb69d5
Begin work on converting urllog.tcl to use an SQLite3 database instead of flat file.
Matti Hamalainen <ccr@tnsp.org>
parents:
8
diff
changeset
|
91 ### SQLite database initialization |
|
e06d41fb69d5
Begin work on converting urllog.tcl to use an SQLite3 database instead of flat file.
Matti Hamalainen <ccr@tnsp.org>
parents:
8
diff
changeset
|
92 if {[catch {sqlite3 urldb $urllog_db_file} uerrmsg]} { |
| 28 | 93 putlog " Could not open SQLite3 database '$urllog_db_file': $uerrmsg" |
| 94 exit 2 | |
|
13
e06d41fb69d5
Begin work on converting urllog.tcl to use an SQLite3 database instead of flat file.
Matti Hamalainen <ccr@tnsp.org>
parents:
8
diff
changeset
|
95 } |
|
e06d41fb69d5
Begin work on converting urllog.tcl to use an SQLite3 database instead of flat file.
Matti Hamalainen <ccr@tnsp.org>
parents:
8
diff
changeset
|
96 |
| 0 | 97 |
| 98 #------------------------------------------------------------------------- | |
| 99 ### Utility functions | |
| 100 proc urllog_log {arg} { | |
|
267
da239a953e24
urllog: Change some setting names, etc.
Matti Hamalainen <ccr@tnsp.org>
parents:
264
diff
changeset
|
101 global urllog_log_enable urllog_name |
| 0 | 102 |
|
267
da239a953e24
urllog: Change some setting names, etc.
Matti Hamalainen <ccr@tnsp.org>
parents:
264
diff
changeset
|
103 if {$urllog_log_enable != 0} { |
| 28 | 104 putlog "$urllog_name: $arg" |
| 105 } | |
| 0 | 106 } |
| 107 | |
| 108 | |
| 152 | 109 proc urllog_ctime {utime} { |
| 28 | 110 if {$utime == "" || $utime == "*"} { |
| 111 set utime 0 | |
| 112 } | |
| 113 return [clock format $utime -format "%d.%m.%Y %H:%M"] | |
| 0 | 114 } |
| 115 | |
| 116 | |
| 117 proc urllog_isnumber {uarg} { | |
| 28 | 118 foreach i [split $uarg {}] { |
|
65
31c8c4f50aa6
urllog: Improve urllog_isnumber function.
Matti Hamalainen <ccr@tnsp.org>
parents:
62
diff
changeset
|
119 if {![string match \[0-9\] $i]} { return 0 } |
| 28 | 120 } |
|
65
31c8c4f50aa6
urllog: Improve urllog_isnumber function.
Matti Hamalainen <ccr@tnsp.org>
parents:
62
diff
changeset
|
121 return 1 |
| 0 | 122 } |
| 123 | |
| 124 | |
| 125 proc urllog_msg {apublic anick achan amsg} { | |
| 28 | 126 global urllog_preferredmsg |
| 0 | 127 |
| 28 | 128 if {$apublic == 1} { |
| 129 putserv "$urllog_preferredmsg $achan :$amsg" | |
| 130 } else { | |
| 131 putserv "$urllog_preferredmsg $anick :$amsg" | |
| 132 } | |
| 0 | 133 } |
| 134 | |
| 135 | |
| 136 proc urllog_verb_msg {anick achan amsg} { | |
| 28 | 137 global urllog_verbose |
| 0 | 138 |
| 28 | 139 if {$urllog_verbose != 0} { |
| 140 urllog_msg 1 $anick $achan $amsg | |
| 141 } | |
| 0 | 142 } |
| 143 | |
| 144 | |
|
116
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
145 proc urllog_sanitize_encoding {uencoding} { |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
146 regsub -- "^\[a-z\]\[a-z\]_\[A-Z\]\[A-Z\]\." $uencoding "" uencoding |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
147 set uencoding [string tolower $uencoding] |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
148 regsub -- "^iso-" $uencoding "iso" uencoding |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
149 return $uencoding |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
150 } |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
151 |
| 294 | 152 |
|
121
bec98a9f8695
Convert the title encoding when outputting to channel.
Matti Hamalainen <ccr@tnsp.org>
parents:
120
diff
changeset
|
153 proc urllog_clean_title {utitle} { |
|
bec98a9f8695
Convert the title encoding when outputting to channel.
Matti Hamalainen <ccr@tnsp.org>
parents:
120
diff
changeset
|
154 if {[catch {set utitle [encoding convertto "iso8859-15" $utitle]} cerrmsg]} { |
|
bec98a9f8695
Convert the title encoding when outputting to channel.
Matti Hamalainen <ccr@tnsp.org>
parents:
120
diff
changeset
|
155 putlog "Could not convert title encoding: $cerrmsg" |
|
bec98a9f8695
Convert the title encoding when outputting to channel.
Matti Hamalainen <ccr@tnsp.org>
parents:
120
diff
changeset
|
156 } |
|
bec98a9f8695
Convert the title encoding when outputting to channel.
Matti Hamalainen <ccr@tnsp.org>
parents:
120
diff
changeset
|
157 return $utitle |
|
bec98a9f8695
Convert the title encoding when outputting to channel.
Matti Hamalainen <ccr@tnsp.org>
parents:
120
diff
changeset
|
158 } |
|
116
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
159 |
| 294 | 160 |
| 0 | 161 #------------------------------------------------------------------------- |
|
150
52350ed97775
urllog: Cleanups, rename/move some global variables.
Matti Hamalainen <ccr@tnsp.org>
parents:
136
diff
changeset
|
162 set urllog_shorturl_str "ABCDEFGHIJKLNMOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789" |
|
13
e06d41fb69d5
Begin work on converting urllog.tcl to use an SQLite3 database instead of flat file.
Matti Hamalainen <ccr@tnsp.org>
parents:
8
diff
changeset
|
163 |
|
150
52350ed97775
urllog: Cleanups, rename/move some global variables.
Matti Hamalainen <ccr@tnsp.org>
parents:
136
diff
changeset
|
164 proc urllog_get_short {utime} { |
|
52350ed97775
urllog: Cleanups, rename/move some global variables.
Matti Hamalainen <ccr@tnsp.org>
parents:
136
diff
changeset
|
165 global urllog_shorturl_prefix urllog_shorturl_str |
|
52350ed97775
urllog: Cleanups, rename/move some global variables.
Matti Hamalainen <ccr@tnsp.org>
parents:
136
diff
changeset
|
166 |
|
52350ed97775
urllog: Cleanups, rename/move some global variables.
Matti Hamalainen <ccr@tnsp.org>
parents:
136
diff
changeset
|
167 set ulen [string length $urllog_shorturl_str] |
| 0 | 168 |
| 28 | 169 set u1 [expr $utime / ($ulen * $ulen)] |
| 170 set utmp [expr $utime % ($ulen * $ulen)] | |
| 171 set u2 [expr $utmp / $ulen] | |
| 172 set u3 [expr $utmp % $ulen] | |
| 0 | 173 |
|
150
52350ed97775
urllog: Cleanups, rename/move some global variables.
Matti Hamalainen <ccr@tnsp.org>
parents:
136
diff
changeset
|
174 return "\[ $urllog_shorturl_prefix[string index $urllog_shorturl_str $u1][string index $urllog_shorturl_str $u2][string index $urllog_shorturl_str $u3] \]" |
|
52350ed97775
urllog: Cleanups, rename/move some global variables.
Matti Hamalainen <ccr@tnsp.org>
parents:
136
diff
changeset
|
175 } |
| 0 | 176 |
| 177 | |
| 178 #------------------------------------------------------------------------- | |
| 179 proc urllog_chop_url {url} { | |
| 28 | 180 global urllog_shorturl_orig |
| 68 | 181 |
| 28 | 182 if {[string length $url] > $urllog_shorturl_orig} { |
| 183 return "[string range $url 0 $urllog_shorturl_orig]..." | |
| 184 } else { | |
| 185 return $url | |
| 186 } | |
| 0 | 187 } |
| 188 | |
| 241 | 189 |
| 0 | 190 #------------------------------------------------------------------------- |
|
83
f171a9fb7b7b
urllog: Split urllog_add function to urllog_exists for checking whether given URL already exists in the database. Use urllog_exists where appropriate.
Matti Hamalainen <ccr@tnsp.org>
parents:
82
diff
changeset
|
191 proc urllog_exists {urlStr urlNick urlHost urlChan} { |
| 28 | 192 global urldb urlmsg_alreadyknown urllog_shorturl |
| 0 | 193 |
|
295
141bb4a2b76f
utillib: utl_escape (which will be deprecated soon).
Matti Hamalainen <ccr@tnsp.org>
parents:
294
diff
changeset
|
194 set usql "SELECT id AS uid, utime AS utime, url AS uurl, user AS uuser, host AS uhost, chan AS uchan, title AS utitle FROM urls WHERE url='[utl_escape $urlStr]'" |
| 297 | 195 urldb eval $usql { |
| 28 | 196 urllog_log "URL said by $urlNick ($urlStr) already known" |
| 197 if {$urllog_shorturl != 0} { | |
|
83
f171a9fb7b7b
urllog: Split urllog_add function to urllog_exists for checking whether given URL already exists in the database. Use urllog_exists where appropriate.
Matti Hamalainen <ccr@tnsp.org>
parents:
82
diff
changeset
|
198 set qstr "[urllog_get_short $uid] " |
| 28 | 199 } else { |
| 200 set qstr "" | |
| 201 } | |
| 202 append qstr "($uuser/$uchan@[urllog_ctime $utime])" | |
|
83
f171a9fb7b7b
urllog: Split urllog_add function to urllog_exists for checking whether given URL already exists in the database. Use urllog_exists where appropriate.
Matti Hamalainen <ccr@tnsp.org>
parents:
82
diff
changeset
|
203 if {[string length $utitle] > 0} { |
|
121
bec98a9f8695
Convert the title encoding when outputting to channel.
Matti Hamalainen <ccr@tnsp.org>
parents:
120
diff
changeset
|
204 set qstr "$urlmsg_alreadyknown - '[urllog_clean_title $utitle]' $qstr" |
| 28 | 205 } else { |
| 206 set qstr "$urlmsg_alreadyknown $qstr" | |
| 207 } | |
| 208 urllog_verb_msg $urlNick $urlChan $qstr | |
| 209 return 0 | |
| 210 } | |
|
83
f171a9fb7b7b
urllog: Split urllog_add function to urllog_exists for checking whether given URL already exists in the database. Use urllog_exists where appropriate.
Matti Hamalainen <ccr@tnsp.org>
parents:
82
diff
changeset
|
211 return 1 |
|
f171a9fb7b7b
urllog: Split urllog_add function to urllog_exists for checking whether given URL already exists in the database. Use urllog_exists where appropriate.
Matti Hamalainen <ccr@tnsp.org>
parents:
82
diff
changeset
|
212 } |
| 0 | 213 |
|
18
1e2232135354
More changes for SQLite support.
Matti Hamalainen <ccr@tnsp.org>
parents:
13
diff
changeset
|
214 |
|
83
f171a9fb7b7b
urllog: Split urllog_add function to urllog_exists for checking whether given URL already exists in the database. Use urllog_exists where appropriate.
Matti Hamalainen <ccr@tnsp.org>
parents:
82
diff
changeset
|
215 #------------------------------------------------------------------------- |
|
f171a9fb7b7b
urllog: Split urllog_add function to urllog_exists for checking whether given URL already exists in the database. Use urllog_exists where appropriate.
Matti Hamalainen <ccr@tnsp.org>
parents:
82
diff
changeset
|
216 proc urllog_addurl {urlStr urlNick urlHost urlChan urlTitle} { |
|
f171a9fb7b7b
urllog: Split urllog_add function to urllog_exists for checking whether given URL already exists in the database. Use urllog_exists where appropriate.
Matti Hamalainen <ccr@tnsp.org>
parents:
82
diff
changeset
|
217 global urldb urllog_shorturl |
|
f171a9fb7b7b
urllog: Split urllog_add function to urllog_exists for checking whether given URL already exists in the database. Use urllog_exists where appropriate.
Matti Hamalainen <ccr@tnsp.org>
parents:
82
diff
changeset
|
218 |
|
93
4e02c0219afe
urllog: Insert NULL into title column when we didn't get a title.
Matti Hamalainen <ccr@tnsp.org>
parents:
92
diff
changeset
|
219 if {$urlTitle == ""} { |
|
4e02c0219afe
urllog: Insert NULL into title column when we didn't get a title.
Matti Hamalainen <ccr@tnsp.org>
parents:
92
diff
changeset
|
220 set uins "NULL" |
|
4e02c0219afe
urllog: Insert NULL into title column when we didn't get a title.
Matti Hamalainen <ccr@tnsp.org>
parents:
92
diff
changeset
|
221 } else { |
|
295
141bb4a2b76f
utillib: utl_escape (which will be deprecated soon).
Matti Hamalainen <ccr@tnsp.org>
parents:
294
diff
changeset
|
222 set uins "'[utl_escape $urlTitle]'" |
|
93
4e02c0219afe
urllog: Insert NULL into title column when we didn't get a title.
Matti Hamalainen <ccr@tnsp.org>
parents:
92
diff
changeset
|
223 } |
|
295
141bb4a2b76f
utillib: utl_escape (which will be deprecated soon).
Matti Hamalainen <ccr@tnsp.org>
parents:
294
diff
changeset
|
224 set usql "INSERT INTO urls (utime,url,user,host,chan,title) VALUES ([unixtime], '[utl_escape $urlStr]', '[utl_escape $urlNick]', '[utl_escape $urlHost]', '[utl_escape $urlChan]', $uins)" |
|
83
f171a9fb7b7b
urllog: Split urllog_add function to urllog_exists for checking whether given URL already exists in the database. Use urllog_exists where appropriate.
Matti Hamalainen <ccr@tnsp.org>
parents:
82
diff
changeset
|
225 if {[catch {urldb eval $usql} uerrmsg]} { |
|
f171a9fb7b7b
urllog: Split urllog_add function to urllog_exists for checking whether given URL already exists in the database. Use urllog_exists where appropriate.
Matti Hamalainen <ccr@tnsp.org>
parents:
82
diff
changeset
|
226 urllog_log "$uerrmsg on SQL:\n$usql" |
| 28 | 227 return 0 |
| 228 } | |
|
82
1bbc79f41a1c
urllog: Rename few variables for clarity.
Matti Hamalainen <ccr@tnsp.org>
parents:
81
diff
changeset
|
229 set uid [urldb last_insert_rowid] |
| 28 | 230 urllog_log "Added URL ($urlNick@$urlChan): $urlStr" |
| 0 | 231 |
| 232 | |
| 28 | 233 ### Let's say something, to confirm that everything went well. |
| 234 if {$urllog_shorturl != 0} { | |
|
82
1bbc79f41a1c
urllog: Rename few variables for clarity.
Matti Hamalainen <ccr@tnsp.org>
parents:
81
diff
changeset
|
235 set qstr "[urllog_get_short $uid] " |
| 28 | 236 } else { |
| 237 set qstr "" | |
| 238 } | |
| 239 if {[string length $urlTitle] > 0} { | |
|
121
bec98a9f8695
Convert the title encoding when outputting to channel.
Matti Hamalainen <ccr@tnsp.org>
parents:
120
diff
changeset
|
240 urllog_verb_msg $urlNick $urlChan "'[urllog_clean_title $urlTitle]' ([urllog_chop_url $urlStr]) $qstr" |
| 28 | 241 } else { |
| 242 urllog_verb_msg $urlNick $urlChan "[urllog_chop_url $urlStr] $qstr" | |
| 243 } | |
| 0 | 244 |
| 28 | 245 return 1 |
| 0 | 246 } |
| 247 | |
| 248 | |
| 249 #------------------------------------------------------------------------- | |
|
251
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
250 proc urllog_dorequest { urlNick urlChan urlStr urlStatus urlSCode urlCode urlData urlMeta } { |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
251 global urlmsg_ioerror urlmsg_timeout urlmsg_errorgettingdoc |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
252 |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
253 upvar 1 $urlStatus ustatus |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
254 upvar 1 $urlSCode uscode |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
255 upvar 1 $urlCode ucode |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
256 upvar 1 $urlData udata |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
257 upvar 1 $urlMeta umeta |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
258 |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
259 if {[catch {set utoken [::http::geturl $urlStr -timeout 6000 -headers {Accept-Encoding identity}]} uerrmsg]} { |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
260 urllog_verb_msg $urlNick $urlChan "$urlmsg_ioerror ($uerrmsg)" |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
261 urllog_log "HTTP request failed: $uerrmsg" |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
262 return 0 |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
263 } |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
264 |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
265 set ustatus [::http::status $utoken] |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
266 if {$ustatus == "timeout"} { |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
267 urllog_verb_msg $urlNick $urlChan "$urlmsg_timeout" |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
268 urllog_log "HTTP request timed out ($urlStr)" |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
269 return 0 |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
270 } |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
271 |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
272 if {$ustatus != "ok"} { |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
273 urllog_verb_msg $urlNick $urlChan "$urlmsg_errorgettingdoc ([::http::error $utoken])" |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
274 urllog_log "Error in HTTP transaction: [::http::error $utoken] ($urlStr)" |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
275 return 0 |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
276 } |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
277 |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
278 set ustatus [::http::status $utoken] |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
279 set uscode [::http::code $utoken] |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
280 set ucode [::http::ncode $utoken] |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
281 set udata [::http::data $utoken] |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
282 array set umeta [::http::meta $utoken] |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
283 ::http::cleanup $utoken |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
284 |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
285 return 1 |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
286 } |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
287 |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
288 #------------------------------------------------------------------------- |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
289 proc urllog_validate_url { urlNick urlChan urlMStr urlMProto } { |
|
302
f487cc166714
urllog: Add message for unknown TLDs.
Matti Hamalainen <ccr@tnsp.org>
parents:
300
diff
changeset
|
290 global urllog_tld_list urlmsg_nosuchhost urllog_httprep urlmsg_unknown_tld |
|
300
2a9ee3f68225
urllog: Make TLD check configurable.
Matti Hamalainen <ccr@tnsp.org>
parents:
299
diff
changeset
|
291 global urllog_shorturl_prefix urllog_shorturl urllog_check_tld |
|
251
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
292 upvar 1 $urlMStr urlStr |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
293 upvar 1 $urlMProto urlProto |
|
3
8003090caa35
Lots of code cleanups, add "fixer" for RasiaTube links (which suck) to point directly to Youtube.
Matti Hamalainen <ccr@tnsp.org>
parents:
0
diff
changeset
|
294 |
|
96
e5a6c27be365
urllog: Comments and cosmetics.
Matti Hamalainen <ccr@tnsp.org>
parents:
95
diff
changeset
|
295 ### Try to guess the URL protocol component (if it is missing) |
| 28 | 296 set u_checktld 1 |
| 297 if {[string match "*www.*" $urlStr] && ![string match "http://*" $urlStr] && ![string match "https://*" $urlStr]} { | |
| 298 set urlStr "http://$urlStr" | |
| 299 } elseif {[string match "*ftp.*" $urlStr] && ![string match "ftp://*" $urlStr]} { | |
| 300 set urlStr "ftp://$urlStr" | |
| 301 } | |
| 0 | 302 |
|
95
687bdd74dfac
urllog: Check if TLS support is enabled when checking if we can fetch title information via HTTP or SSL/HTTP.
Matti Hamalainen <ccr@tnsp.org>
parents:
93
diff
changeset
|
303 ### Handle URLs that have an IPv4-address |
|
251
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
304 if {[regexp "(\[a-z\]+)://(\[0-9\]{1,3})\\.(\[0-9\]{1,3})\\.(\[0-9\]{1,3})\\.(\[0-9\]{1,3})" $urlStr u_match urlProto ni1 ni2 ni3 ni4]} { |
| 28 | 305 # Check if the IP is on local network |
|
92
f6f4595856ff
urllog: Cosmetics. Remove useless parenthesis.
Matti Hamalainen <ccr@tnsp.org>
parents:
91
diff
changeset
|
306 if {$ni1 == 127 || $ni1 == 10 || ($ni1 == 192 && $ni2 == 168) || $ni1 == 0} { |
| 28 | 307 urllog_log "URL pointing to local or invalid network, ignored ($urlStr)." |
| 308 return 0 | |
| 309 } | |
| 310 # Skip TLD check for URLs with IP address | |
| 311 set u_checktld 0 | |
| 312 } | |
| 0 | 313 |
|
96
e5a6c27be365
urllog: Comments and cosmetics.
Matti Hamalainen <ccr@tnsp.org>
parents:
95
diff
changeset
|
314 ### Check now if we have an ShortURL here ... |
|
150
52350ed97775
urllog: Cleanups, rename/move some global variables.
Matti Hamalainen <ccr@tnsp.org>
parents:
136
diff
changeset
|
315 if {[string match "$urllog_shorturl_prefix*" $urlStr]} { |
|
98
fbbe7ee40e2f
urllog: Improve one informational / error message.
Matti Hamalainen <ccr@tnsp.org>
parents:
97
diff
changeset
|
316 urllog_log "Ignoring ShortURL from $urlNick: $urlStr" |
|
252
eb2fce89b8ab
urllog: Comment out some currently unused code.
Matti Hamalainen <ccr@tnsp.org>
parents:
251
diff
changeset
|
317 # set uud "" |
|
eb2fce89b8ab
urllog: Comment out some currently unused code.
Matti Hamalainen <ccr@tnsp.org>
parents:
251
diff
changeset
|
318 # set usql "SELECT id AS uid, url AS uurl, user AS uuser, host AS uhost, chan AS uchan, title AS utitle FROM urls WHERE utime=$uud" |
|
eb2fce89b8ab
urllog: Comment out some currently unused code.
Matti Hamalainen <ccr@tnsp.org>
parents:
251
diff
changeset
|
319 # urldb eval $usql { |
|
eb2fce89b8ab
urllog: Comment out some currently unused code.
Matti Hamalainen <ccr@tnsp.org>
parents:
251
diff
changeset
|
320 # urllog_verb_msg $urlNick $urlChan "'$utitle' - $uurl" |
|
eb2fce89b8ab
urllog: Comment out some currently unused code.
Matti Hamalainen <ccr@tnsp.org>
parents:
251
diff
changeset
|
321 # return 1 |
|
eb2fce89b8ab
urllog: Comment out some currently unused code.
Matti Hamalainen <ccr@tnsp.org>
parents:
251
diff
changeset
|
322 # } |
| 28 | 323 return 0 |
| 324 } | |
| 0 | 325 |
|
95
687bdd74dfac
urllog: Check if TLS support is enabled when checking if we can fetch title information via HTTP or SSL/HTTP.
Matti Hamalainen <ccr@tnsp.org>
parents:
93
diff
changeset
|
326 ### Get URL protocol component |
|
251
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
327 set urlProto "" |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
328 regexp "(\[a-z\]+)://" $urlStr u_match urlProto |
|
95
687bdd74dfac
urllog: Check if TLS support is enabled when checking if we can fetch title information via HTTP or SSL/HTTP.
Matti Hamalainen <ccr@tnsp.org>
parents:
93
diff
changeset
|
329 |
| 28 | 330 ### Check the PORT (if the ":" is there) |
| 331 set u_record [split $urlStr "/"] | |
| 332 set u_hostname [lindex $u_record 2] | |
| 333 set u_port [lindex [split $u_hostname ":"] end] | |
| 0 | 334 |
| 28 | 335 if {![urllog_isnumber $u_port] && $u_port != "" && $u_port != $u_hostname} { |
| 336 urllog_log "Broken URL from $urlNick: ($urlStr) illegal port $u_port" | |
| 337 return 0 | |
| 338 } | |
| 0 | 339 |
| 298 | 340 ### Is it a http or ftp url? |
|
251
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
341 if {$urlProto != "http" && $urlProto != "https" && $urlProto != "ftp"} { |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
342 urllog_log "Broken URL from $urlNick: ($urlStr) UNSUPPORTED protocol class ($urlProto)." |
| 28 | 343 return 0 |
| 344 } | |
| 0 | 345 |
| 28 | 346 ### Check the Top Level Domain (TLD) validity |
|
300
2a9ee3f68225
urllog: Make TLD check configurable.
Matti Hamalainen <ccr@tnsp.org>
parents:
299
diff
changeset
|
347 if {$urllog_check_tld != 0 && $u_checktld != 0} { |
| 28 | 348 set u_sane [lindex [split $u_hostname "."] end] |
| 349 set u_tld [lindex [split $u_sane ":"] 0] | |
| 350 set u_found 0 | |
| 0 | 351 |
| 28 | 352 if {[string length $u_tld] == 2} { |
| 353 # Assume all 2-letter domains to be valid :) | |
| 354 set u_found 1 | |
| 355 } else { | |
| 356 # Check our list of known TLDs | |
|
300
2a9ee3f68225
urllog: Make TLD check configurable.
Matti Hamalainen <ccr@tnsp.org>
parents:
299
diff
changeset
|
357 foreach itld $urllog_tld_list { |
| 28 | 358 if {[string match $itld $u_tld]} { |
| 359 set u_found 1 | |
| 360 } | |
| 361 } | |
| 362 } | |
| 0 | 363 |
| 28 | 364 if {$u_found == 0} { |
|
302
f487cc166714
urllog: Add message for unknown TLDs.
Matti Hamalainen <ccr@tnsp.org>
parents:
300
diff
changeset
|
365 urllog_log "Broken URL from $urlNick: ($urlStr) unknown TLD: ${u_tld}." |
|
f487cc166714
urllog: Add message for unknown TLDs.
Matti Hamalainen <ccr@tnsp.org>
parents:
300
diff
changeset
|
366 urllog_verb_msg $urlNick $urlChan $urlmsg_unknown_tld |
| 28 | 367 return 0 |
| 368 } | |
| 369 } | |
| 0 | 370 |
| 28 | 371 set urlStr [string map $urllog_httprep $urlStr] |
|
251
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
372 return 1 |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
373 } |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
374 |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
375 |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
376 #------------------------------------------------------------------------- |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
377 proc urllog_check_url {urlStr urlNick urlHost urlChan} { |
|
299
1ff281e821a3
urllog: Make rasiatube hack configurable.
Matti Hamalainen <ccr@tnsp.org>
parents:
298
diff
changeset
|
378 global urllog_encoding http_tls_support urlmsg_errorgettingdoc |
|
304
f1589fe20732
urllog: Added urllog_extra_strict option.
Matti Hamalainen <ccr@tnsp.org>
parents:
302
diff
changeset
|
379 global urllog_extra_checks urllog_extra_strict urllog_rasiatube_hack |
|
3
8003090caa35
Lots of code cleanups, add "fixer" for RasiaTube links (which suck) to point directly to Youtube.
Matti Hamalainen <ccr@tnsp.org>
parents:
0
diff
changeset
|
380 |
|
91
6f4bfd8e9447
urllog: Reorder code and make it simpler by removing duplicate checks.
Matti Hamalainen <ccr@tnsp.org>
parents:
90
diff
changeset
|
381 ### Does the URL already exist? |
|
6f4bfd8e9447
urllog: Reorder code and make it simpler by removing duplicate checks.
Matti Hamalainen <ccr@tnsp.org>
parents:
90
diff
changeset
|
382 if {![urllog_exists $urlStr $urlNick $urlHost $urlChan]} { |
|
6f4bfd8e9447
urllog: Reorder code and make it simpler by removing duplicate checks.
Matti Hamalainen <ccr@tnsp.org>
parents:
90
diff
changeset
|
383 return 1 |
|
6f4bfd8e9447
urllog: Reorder code and make it simpler by removing duplicate checks.
Matti Hamalainen <ccr@tnsp.org>
parents:
90
diff
changeset
|
384 } |
|
251
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
385 |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
386 ### Validate URL compoments, etc. |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
387 set u_proto "" |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
388 if {![urllog_validate_url $urlNick $urlChan urlStr u_proto]} { |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
389 return 1 |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
390 } |
| 0 | 391 |
|
267
da239a953e24
urllog: Change some setting names, etc.
Matti Hamalainen <ccr@tnsp.org>
parents:
264
diff
changeset
|
392 ### Do we perform additional checks? |
|
da239a953e24
urllog: Change some setting names, etc.
Matti Hamalainen <ccr@tnsp.org>
parents:
264
diff
changeset
|
393 if {$urllog_extra_checks == 0 || !(($http_tls_support != 0 && $u_proto == "https") || $u_proto == "http")} { |
| 230 | 394 # No optional checks, or it's not http/https. |
| 306 | 395 if {$urllog_extra_strict == 0} { |
|
304
f1589fe20732
urllog: Added urllog_extra_strict option.
Matti Hamalainen <ccr@tnsp.org>
parents:
302
diff
changeset
|
396 # Strict checking disabled, so add the URL, if it does not exist already. |
|
f1589fe20732
urllog: Added urllog_extra_strict option.
Matti Hamalainen <ccr@tnsp.org>
parents:
302
diff
changeset
|
397 urllog_addurl $urlStr $urlNick $urlHost $urlChan "" |
|
f1589fe20732
urllog: Added urllog_extra_strict option.
Matti Hamalainen <ccr@tnsp.org>
parents:
302
diff
changeset
|
398 return 1 |
| 306 | 399 } elseif {$http_tls_support == 0 && $u_proto == "https"} { |
|
304
f1589fe20732
urllog: Added urllog_extra_strict option.
Matti Hamalainen <ccr@tnsp.org>
parents:
302
diff
changeset
|
400 # Strict ENABLED: If TLS support is disabled and we have https, do nothing |
|
f1589fe20732
urllog: Added urllog_extra_strict option.
Matti Hamalainen <ccr@tnsp.org>
parents:
302
diff
changeset
|
401 return 1 |
| 306 | 402 } elseif {$u_proto != "http" && $u_prot != "https"} { |
|
304
f1589fe20732
urllog: Added urllog_extra_strict option.
Matti Hamalainen <ccr@tnsp.org>
parents:
302
diff
changeset
|
403 # Strict ENABLED: It's not http, or https |
|
f1589fe20732
urllog: Added urllog_extra_strict option.
Matti Hamalainen <ccr@tnsp.org>
parents:
302
diff
changeset
|
404 return 1 |
|
f1589fe20732
urllog: Added urllog_extra_strict option.
Matti Hamalainen <ccr@tnsp.org>
parents:
302
diff
changeset
|
405 } |
| 28 | 406 } |
|
7
50b52294e93e
urllog: Strip ‏ entities from titles; Some work on SSL/https support.
Matti Hamalainen <ccr@tnsp.org>
parents:
4
diff
changeset
|
407 |
| 28 | 408 ### Does the document pointed by the URL exist? |
|
251
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
409 if {![urllog_dorequest $urlNick $urlChan $urlStr ustatus uscode ucode udata umeta]} { |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
410 return 1 |
| 28 | 411 } |
| 0 | 412 |
|
251
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
413 ### Handle redirects |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
414 if {$ucode >= 301 && $ucode <= 302} { |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
415 set nurlStr $umeta(Location) |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
416 urllog_log "Redirection: $urlStr -> $nurlStr" |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
417 set urlStr $nurlStr |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
418 |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
419 if {![urllog_validate_url $urlNick $urlChan urlStr urlProto]} { |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
420 return 1 |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
421 } |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
422 |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
423 if {![urllog_dorequest $urlNick $urlChan $urlStr ustatus uscode ucode udata umeta]} { |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
424 return 1 |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
425 } |
| 28 | 426 } |
|
3
8003090caa35
Lots of code cleanups, add "fixer" for RasiaTube links (which suck) to point directly to Youtube.
Matti Hamalainen <ccr@tnsp.org>
parents:
0
diff
changeset
|
427 |
|
251
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
428 ### Handle 2nd level redirects |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
429 if {$ucode >= 301 && $ucode <= 302} { |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
430 set nurlStr $umeta(Location) |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
431 urllog_log "Redirection #2: $urlStr -> $nurlStr" |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
432 set urlStr $nurlStr |
|
116
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
433 |
|
251
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
434 if {![urllog_validate_url $urlNick $urlChan urlStr urlProto]} { |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
435 return 1 |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
436 } |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
437 |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
438 if {![urllog_dorequest $urlNick $urlChan $urlStr ustatus uscode ucode udata umeta]} { |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
439 return 1 |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
440 } |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
441 } |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
442 |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
443 # Final document |
|
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
444 if {$ucode >= 200 && $ucode <= 205} { |
|
116
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
445 set uenc_doc "" |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
446 set uenc_http "" |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
447 set uencoding "" |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
448 |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
449 # Get information about specified character encodings |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
450 if {[info exists umeta(Content-Type)] && [regexp -nocase {charset\s*=\s*([a-z0-9._-]+)} $umeta(Content-Type) umatches uenc_http]} { |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
451 # Found character set encoding information in HTTP headers |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
452 } |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
453 |
|
150
52350ed97775
urllog: Cleanups, rename/move some global variables.
Matti Hamalainen <ccr@tnsp.org>
parents:
136
diff
changeset
|
454 if {[regexp -nocase -- "<meta.\*\?content=\"text/html.\*\?charset=(\[^\"\]*)\".\*\?/\?>" $udata umatches uenc_doc]} { |
|
116
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
455 # Found old style HTML meta tag with character set information |
|
150
52350ed97775
urllog: Cleanups, rename/move some global variables.
Matti Hamalainen <ccr@tnsp.org>
parents:
136
diff
changeset
|
456 } elseif {[regexp -nocase -- "<meta.\*\?charset=\"(\[^\"\]*)\".\*\?/\?>" $udata umatches uenc_doc]} { |
|
116
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
457 # Found HTML5 style meta tag with character set information |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
458 } |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
459 |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
460 # Make sanitized versions of the encoding strings |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
461 set uenc_http2 [urllog_sanitize_encoding $uenc_http] |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
462 set uenc_doc2 [urllog_sanitize_encoding $uenc_doc] |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
463 |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
464 # KLUDGE! |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
465 set uencoding $uenc_http2 |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
466 |
|
210
52cadf5a12b6
urllog: Disable some debug logging.
Matti Hamalainen <ccr@tnsp.org>
parents:
209
diff
changeset
|
467 # putlog "got charsets : http='$uenc_http', doc='$uenc_doc' / sanitized http='$uenc_http2', doc='$uenc_doc2'" |
|
150
52350ed97775
urllog: Cleanups, rename/move some global variables.
Matti Hamalainen <ccr@tnsp.org>
parents:
136
diff
changeset
|
468 |
|
52350ed97775
urllog: Cleanups, rename/move some global variables.
Matti Hamalainen <ccr@tnsp.org>
parents:
136
diff
changeset
|
469 |
|
116
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
470 # Check if the document has specified encoding |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
471 if {$uenc_doc != ""} { |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
472 # Does it differ from what HTTP says? |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
473 if {$uenc_http != "" && $uenc_doc != $uenc_http && $uenc_doc2 != $uenc_http2} { |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
474 # Yes, we will try reconverting |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
475 set uencoding $uenc_doc2 |
| 28 | 476 } |
|
116
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
477 } elseif {$uenc_http == ""} { |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
478 # If _NO_ known encoding of any kind, assume the default of iso8859-1 |
|
86
4c2b6482c08c
urllog: Different strategy for charset encoding conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
84
diff
changeset
|
479 set uencoding "iso8859-1" |
|
4c2b6482c08c
urllog: Different strategy for charset encoding conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
84
diff
changeset
|
480 } |
| 0 | 481 |
|
116
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
482 # Get the document title, if any |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
483 set urlTitle "" |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
484 if {[regexp -nocase -- "<title>(.\*\?)</title>" $udata umatches urlTitle]} { |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
485 # If character set conversion is required, do it now |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
486 if {$uencoding != ""} { |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
487 if {[catch {set urlTitle [encoding convertfrom $uencoding $urlTitle]} cerrmsg]} { |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
488 urllog_log "Error in charset conversion: $cerrmsg" |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
489 } |
| 28 | 490 } |
|
150
52350ed97775
urllog: Cleanups, rename/move some global variables.
Matti Hamalainen <ccr@tnsp.org>
parents:
136
diff
changeset
|
491 |
|
116
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
492 # Convert some HTML entities to plaintext and do some cleanup |
|
291
54d34d086b47
urllog: Use the utility lib for entity conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
289
diff
changeset
|
493 set utmp [utl_convert_html_ent $urlTitle] |
|
116
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
494 regsub -all "\r|\n|\t" $utmp " " utmp |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
495 regsub -all " *" $utmp " " utmp |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
496 set urlTitle [string trim $utmp] |
| 28 | 497 } |
|
3
8003090caa35
Lots of code cleanups, add "fixer" for RasiaTube links (which suck) to point directly to Youtube.
Matti Hamalainen <ccr@tnsp.org>
parents:
0
diff
changeset
|
498 |
| 28 | 499 # Rasiatube hack |
|
299
1ff281e821a3
urllog: Make rasiatube hack configurable.
Matti Hamalainen <ccr@tnsp.org>
parents:
298
diff
changeset
|
500 if {$urllog_rasiatube_hack != 0 && [string match "*/rasiatube/view*" $urlStr]} { |
| 28 | 501 set rasia 0 |
|
118
e5f2961a6145
urllog: Improve rasiatube URL de-mangling.
Matti Hamalainen <ccr@tnsp.org>
parents:
117
diff
changeset
|
502 if {[regexp -nocase -- "<link rel=\"video_src\"\.\*\?file=(http://\[^&\]+)&" $udata umatches utmp]} { |
|
e5f2961a6145
urllog: Improve rasiatube URL de-mangling.
Matti Hamalainen <ccr@tnsp.org>
parents:
117
diff
changeset
|
503 regsub -all "\/v\/" $utmp "\/watch\?v=" urlStr |
| 28 | 504 set rasia 1 |
| 505 } else { | |
|
118
e5f2961a6145
urllog: Improve rasiatube URL de-mangling.
Matti Hamalainen <ccr@tnsp.org>
parents:
117
diff
changeset
|
506 if {[regexp -nocase -- "SWFObject.\"(\[^\"\]+)\", *\"flashvideo" $udata umatches utmp]} { |
|
e5f2961a6145
urllog: Improve rasiatube URL de-mangling.
Matti Hamalainen <ccr@tnsp.org>
parents:
117
diff
changeset
|
507 regsub "http:\/\/www.dailymotion.com\/swf\/" $utmp "http:\/\/www.dailymotion.com\/video\/" urlStr |
| 28 | 508 set rasia 1 |
| 509 } | |
| 510 } | |
| 511 if {$rasia != 0} { | |
| 512 urllog_log "RasiaTube mangler: $urlStr" | |
| 513 urllog_verb_msg $urlNick $urlChan "Korjataan haiseva rasiatube-linkki: $urlStr" | |
| 514 } | |
| 515 } | |
|
3
8003090caa35
Lots of code cleanups, add "fixer" for RasiaTube links (which suck) to point directly to Youtube.
Matti Hamalainen <ccr@tnsp.org>
parents:
0
diff
changeset
|
516 |
|
83
f171a9fb7b7b
urllog: Split urllog_add function to urllog_exists for checking whether given URL already exists in the database. Use urllog_exists where appropriate.
Matti Hamalainen <ccr@tnsp.org>
parents:
82
diff
changeset
|
517 # Check if the URL already exists, just in case we had some redirects |
|
f171a9fb7b7b
urllog: Split urllog_add function to urllog_exists for checking whether given URL already exists in the database. Use urllog_exists where appropriate.
Matti Hamalainen <ccr@tnsp.org>
parents:
82
diff
changeset
|
518 if {[urllog_exists $urlStr $urlNick $urlHost $urlChan]} { |
|
f171a9fb7b7b
urllog: Split urllog_add function to urllog_exists for checking whether given URL already exists in the database. Use urllog_exists where appropriate.
Matti Hamalainen <ccr@tnsp.org>
parents:
82
diff
changeset
|
519 urllog_addurl $urlStr $urlNick $urlHost $urlChan $urlTitle |
|
f171a9fb7b7b
urllog: Split urllog_add function to urllog_exists for checking whether given URL already exists in the database. Use urllog_exists where appropriate.
Matti Hamalainen <ccr@tnsp.org>
parents:
82
diff
changeset
|
520 } |
| 28 | 521 return 1 |
| 522 } else { | |
|
116
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
523 urllog_verb_msg $urlNick $urlChan "$urlmsg_errorgettingdoc ($ucode)" |
|
224
aaf433ab696a
urllog: Improve error messages a bit.
Matti Hamalainen <ccr@tnsp.org>
parents:
223
diff
changeset
|
524 urllog_log "Error fetching document: status=$ustatus, code=$ucode, scode=$uscode, url=$urlStr" |
| 28 | 525 } |
| 0 | 526 } |
| 527 | |
| 528 | |
| 529 #------------------------------------------------------------------------- | |
|
219
4e09bcc48851
urllog: Add settings for specifying channels where URL logging is active, and where !urlfind functionality works (separately, if so desired.)
Matti Hamalainen <ccr@tnsp.org>
parents:
218
diff
changeset
|
530 |
|
4e09bcc48851
urllog: Add settings for specifying channels where URL logging is active, and where !urlfind functionality works (separately, if so desired.)
Matti Hamalainen <ccr@tnsp.org>
parents:
218
diff
changeset
|
531 |
| 249 | 532 proc urllog_check_line {unick uhost uhand uchan utext} { |
|
219
4e09bcc48851
urllog: Add settings for specifying channels where URL logging is active, and where !urlfind functionality works (separately, if so desired.)
Matti Hamalainen <ccr@tnsp.org>
parents:
218
diff
changeset
|
533 global urllog_log_channels |
|
4e09bcc48851
urllog: Add settings for specifying channels where URL logging is active, and where !urlfind functionality works (separately, if so desired.)
Matti Hamalainen <ccr@tnsp.org>
parents:
218
diff
changeset
|
534 |
| 28 | 535 ### Check the nick |
| 87 | 536 if {$unick == "*"} { |
| 249 | 537 urllog_log "urllog_check_line: Nick was wc, this should not happen." |
| 28 | 538 return 0 |
| 539 } | |
| 0 | 540 |
|
219
4e09bcc48851
urllog: Add settings for specifying channels where URL logging is active, and where !urlfind functionality works (separately, if so desired.)
Matti Hamalainen <ccr@tnsp.org>
parents:
218
diff
changeset
|
541 ### Check the channel |
| 229 | 542 foreach akey [split $urllog_log_channels ";"] { |
|
219
4e09bcc48851
urllog: Add settings for specifying channels where URL logging is active, and where !urlfind functionality works (separately, if so desired.)
Matti Hamalainen <ccr@tnsp.org>
parents:
218
diff
changeset
|
543 if {[string match $akey $uchan]} { |
|
4e09bcc48851
urllog: Add settings for specifying channels where URL logging is active, and where !urlfind functionality works (separately, if so desired.)
Matti Hamalainen <ccr@tnsp.org>
parents:
218
diff
changeset
|
544 ### Do the URL checking |
|
4e09bcc48851
urllog: Add settings for specifying channels where URL logging is active, and where !urlfind functionality works (separately, if so desired.)
Matti Hamalainen <ccr@tnsp.org>
parents:
218
diff
changeset
|
545 foreach str [split $utext " "] { |
|
221
b8bf9d7666b6
urllog: Improve URL / link matching.
Matti Hamalainen <ccr@tnsp.org>
parents:
219
diff
changeset
|
546 if {[regexp "((ftp|http|https)://\[^\[:space:\]\]+|^(www|ftp)\.\[^\[:space:\]\]+)" $str ulink]} { |
|
251
e59f0c3ea0f4
urllog: Handle first and second level redirects.
Matti Hamalainen <ccr@tnsp.org>
parents:
250
diff
changeset
|
547 urllog_check_url $str $unick $uhost $uchan |
|
219
4e09bcc48851
urllog: Add settings for specifying channels where URL logging is active, and where !urlfind functionality works (separately, if so desired.)
Matti Hamalainen <ccr@tnsp.org>
parents:
218
diff
changeset
|
548 } |
|
4e09bcc48851
urllog: Add settings for specifying channels where URL logging is active, and where !urlfind functionality works (separately, if so desired.)
Matti Hamalainen <ccr@tnsp.org>
parents:
218
diff
changeset
|
549 } |
|
4e09bcc48851
urllog: Add settings for specifying channels where URL logging is active, and where !urlfind functionality works (separately, if so desired.)
Matti Hamalainen <ccr@tnsp.org>
parents:
218
diff
changeset
|
550 return 0 |
| 28 | 551 } |
| 552 } | |
| 0 | 553 |
| 28 | 554 return 0 |
| 0 | 555 } |
| 556 | |
| 557 | |
| 558 #------------------------------------------------------------------------- | |
| 559 ### Parse arguments, find and show the results | |
| 560 proc urllog_find {unick uhand uchan utext upublic} { | |
|
62
6428b1bcb34b
urllog: Remove some global variable references where they are not used.
Matti Hamalainen <ccr@tnsp.org>
parents:
50
diff
changeset
|
561 global urllog_shorturl urldb |
| 28 | 562 global urllog_showmax_pub urllog_showmax_priv urlmsg_nomatch |
| 0 | 563 |
| 28 | 564 if {$upublic == 0} { |
| 565 set ulimit 5 | |
| 566 } else { | |
| 567 set ulimit 3 | |
| 568 } | |
|
19
9cf22053e5da
Repair !urlfind functionality.
Matti Hamalainen <ccr@tnsp.org>
parents:
18
diff
changeset
|
569 |
| 28 | 570 ### Parse the given command |
| 571 urllog_log "$unick/$uhand searched URL: $utext" | |
| 0 | 572 |
| 28 | 573 set ftokens [split $utext " "] |
| 574 set fpatlist "" | |
| 575 foreach ftoken $ftokens { | |
| 576 set fprefix [string range $ftoken 0 0] | |
| 577 set fpattern [string range $ftoken 1 end] | |
|
295
141bb4a2b76f
utillib: utl_escape (which will be deprecated soon).
Matti Hamalainen <ccr@tnsp.org>
parents:
294
diff
changeset
|
578 set qpattern "'%[utl_escape $fpattern]%'" |
| 0 | 579 |
| 28 | 580 if {$fprefix == "-"} { |
|
128
0d21b9d1d2b9
urllog: Improve search functionality.
Matti Hamalainen <ccr@tnsp.org>
parents:
127
diff
changeset
|
581 lappend fpatlist "(url NOT LIKE $qpattern OR title NOT LIKE $qpattern)" |
| 28 | 582 } elseif {$fprefix == "%"} { |
|
128
0d21b9d1d2b9
urllog: Improve search functionality.
Matti Hamalainen <ccr@tnsp.org>
parents:
127
diff
changeset
|
583 lappend fpatlist "user LIKE $qpattern" |
| 28 | 584 } elseif {$fprefix == "@"} { |
| 585 # foo | |
|
112
fae3dd7a8b20
urllog: Oops, a typo in variable name. Fixed.
Matti Hamalainen <ccr@tnsp.org>
parents:
111
diff
changeset
|
586 } elseif {$fprefix == "+"} { |
|
128
0d21b9d1d2b9
urllog: Improve search functionality.
Matti Hamalainen <ccr@tnsp.org>
parents:
127
diff
changeset
|
587 lappend fpatlist "(url LIKE $qpattern OR title LIKE $qpattern)" |
| 28 | 588 } else { |
|
295
141bb4a2b76f
utillib: utl_escape (which will be deprecated soon).
Matti Hamalainen <ccr@tnsp.org>
parents:
294
diff
changeset
|
589 set qpattern "'%[utl_escape $ftoken]%'" |
|
128
0d21b9d1d2b9
urllog: Improve search functionality.
Matti Hamalainen <ccr@tnsp.org>
parents:
127
diff
changeset
|
590 lappend fpatlist "(url LIKE $qpattern OR title LIKE $qpattern)" |
| 28 | 591 } |
| 592 } | |
|
19
9cf22053e5da
Repair !urlfind functionality.
Matti Hamalainen <ccr@tnsp.org>
parents:
18
diff
changeset
|
593 |
|
27
6e381916b016
Some fixes in the query mechanisms of QuoteDB and URLLog.
Matti Hamalainen <ccr@tnsp.org>
parents:
20
diff
changeset
|
594 if {[llength $fpatlist] > 0} { |
|
6e381916b016
Some fixes in the query mechanisms of QuoteDB and URLLog.
Matti Hamalainen <ccr@tnsp.org>
parents:
20
diff
changeset
|
595 set fquery "WHERE [join $fpatlist " AND "]" |
|
6e381916b016
Some fixes in the query mechanisms of QuoteDB and URLLog.
Matti Hamalainen <ccr@tnsp.org>
parents:
20
diff
changeset
|
596 } else { |
|
6e381916b016
Some fixes in the query mechanisms of QuoteDB and URLLog.
Matti Hamalainen <ccr@tnsp.org>
parents:
20
diff
changeset
|
597 set fquery "" |
|
6e381916b016
Some fixes in the query mechanisms of QuoteDB and URLLog.
Matti Hamalainen <ccr@tnsp.org>
parents:
20
diff
changeset
|
598 } |
| 68 | 599 |
| 28 | 600 set iresults 0 |
|
82
1bbc79f41a1c
urllog: Rename few variables for clarity.
Matti Hamalainen <ccr@tnsp.org>
parents:
81
diff
changeset
|
601 set usql "SELECT id AS uid, utime AS utime, url AS uurl, user AS uuser, host AS uhost FROM urls $fquery ORDER BY utime DESC LIMIT $ulimit" |
| 68 | 602 urldb eval $usql { |
| 28 | 603 incr iresults |
| 604 set shortURL $uurl | |
|
82
1bbc79f41a1c
urllog: Rename few variables for clarity.
Matti Hamalainen <ccr@tnsp.org>
parents:
81
diff
changeset
|
605 if {$urllog_shorturl != 0 && $uid != ""} { |
|
1bbc79f41a1c
urllog: Rename few variables for clarity.
Matti Hamalainen <ccr@tnsp.org>
parents:
81
diff
changeset
|
606 set shortURL "$shortURL [urllog_get_short $uid]" |
| 28 | 607 } |
| 608 urllog_msg $upublic $unick $uchan "#$iresults: $shortURL ($uuser@[urllog_ctime $utime])" | |
| 609 } | |
| 610 | |
| 611 if {$iresults == 0} { | |
| 612 # If no URLs were found | |
| 613 urllog_msg $upublic $unick $uchan $urlmsg_nomatch | |
| 614 } | |
| 0 | 615 |
| 28 | 616 return 0 |
| 0 | 617 } |
| 618 | |
| 619 | |
| 620 #------------------------------------------------------------------------- | |
| 621 ### Finding binded functions | |
| 622 proc urllog_pub_urlfind {unick uhost uhand uchan utext} { | |
|
219
4e09bcc48851
urllog: Add settings for specifying channels where URL logging is active, and where !urlfind functionality works (separately, if so desired.)
Matti Hamalainen <ccr@tnsp.org>
parents:
218
diff
changeset
|
623 global urllog_search_channels |
|
4e09bcc48851
urllog: Add settings for specifying channels where URL logging is active, and where !urlfind functionality works (separately, if so desired.)
Matti Hamalainen <ccr@tnsp.org>
parents:
218
diff
changeset
|
624 |
|
4e09bcc48851
urllog: Add settings for specifying channels where URL logging is active, and where !urlfind functionality works (separately, if so desired.)
Matti Hamalainen <ccr@tnsp.org>
parents:
218
diff
changeset
|
625 foreach akey [split $urllog_search_channels ";"] { |
|
4e09bcc48851
urllog: Add settings for specifying channels where URL logging is active, and where !urlfind functionality works (separately, if so desired.)
Matti Hamalainen <ccr@tnsp.org>
parents:
218
diff
changeset
|
626 if {[string match $akey $uchan]} { |
|
4e09bcc48851
urllog: Add settings for specifying channels where URL logging is active, and where !urlfind functionality works (separately, if so desired.)
Matti Hamalainen <ccr@tnsp.org>
parents:
218
diff
changeset
|
627 return [urllog_find $unick $uhand $uchan $utext 1] |
|
4e09bcc48851
urllog: Add settings for specifying channels where URL logging is active, and where !urlfind functionality works (separately, if so desired.)
Matti Hamalainen <ccr@tnsp.org>
parents:
218
diff
changeset
|
628 } |
|
4e09bcc48851
urllog: Add settings for specifying channels where URL logging is active, and where !urlfind functionality works (separately, if so desired.)
Matti Hamalainen <ccr@tnsp.org>
parents:
218
diff
changeset
|
629 } |
| 28 | 630 return 0 |
| 0 | 631 } |
| 632 | |
| 633 | |
| 634 proc urllog_msg_urlfind {unick uhost uhand utext} { | |
| 28 | 635 urllog_find $unick $uhand "" $utext 0 |
| 636 return 0 | |
|
3
8003090caa35
Lots of code cleanups, add "fixer" for RasiaTube links (which suck) to point directly to Youtube.
Matti Hamalainen <ccr@tnsp.org>
parents:
0
diff
changeset
|
637 } |
| 0 | 638 |
| 639 # end of script |