Mercurial > hg > egg-tcls
annotate urllog.tcl @ 224:aaf433ab696a
urllog: Improve error messages a bit.
| author | Matti Hamalainen <ccr@tnsp.org> |
|---|---|
| date | Thu, 30 Oct 2014 15:48:43 +0200 |
| parents | 606c2a48b2ce |
| children | cb86368b8fcd |
| rev | line source |
|---|---|
| 0 | 1 ########################################################################## |
| 2 # | |
| 222 | 3 # URLLog v2.3.0 by Matti 'ccr' Hamalainen <ccr@tnsp.org> |
|
176
eda776bcb7ed
urllog: Bump copyright, and version.
Matti Hamalainen <ccr@tnsp.org>
parents:
160
diff
changeset
|
4 # (C) Copyright 2000-2014 Tecnic Software productions (TNSP) |
| 0 | 5 # |
|
113
077c7383f36f
urllog: Add line about the script's license.
Matti Hamalainen <ccr@tnsp.org>
parents:
112
diff
changeset
|
6 # This script is freely distributable under GNU GPL (version 2) license. |
|
077c7383f36f
urllog: Add line about the script's license.
Matti Hamalainen <ccr@tnsp.org>
parents:
112
diff
changeset
|
7 # |
| 0 | 8 ########################################################################## |
| 9 # | |
|
50
f69363fc1f61
Update some comments and add a bit of documentation.
Matti Hamalainen <ccr@tnsp.org>
parents:
49
diff
changeset
|
10 # URL-logger script for EggDrop IRC robot, utilizing SQLite3 database |
|
81
17e542b7985a
urllog, quotedb: Improve documentation.
Matti Hamalainen <ccr@tnsp.org>
parents:
73
diff
changeset
|
11 # This script requires SQLite TCL extension. Under Debian, you need: |
|
17e542b7985a
urllog, quotedb: Improve documentation.
Matti Hamalainen <ccr@tnsp.org>
parents:
73
diff
changeset
|
12 # tcl8.5 libsqlite3-tcl (and eggdrop eggdrop-data, of course) |
|
50
f69363fc1f61
Update some comments and add a bit of documentation.
Matti Hamalainen <ccr@tnsp.org>
parents:
49
diff
changeset
|
13 # |
|
81
17e542b7985a
urllog, quotedb: Improve documentation.
Matti Hamalainen <ccr@tnsp.org>
parents:
73
diff
changeset
|
14 # NOTICE! If you are upgrading to URLLog v2.0+ from any 1.x version, you |
|
50
f69363fc1f61
Update some comments and add a bit of documentation.
Matti Hamalainen <ccr@tnsp.org>
parents:
49
diff
changeset
|
15 # may want to run a conversion script against your URL-database file, |
|
f69363fc1f61
Update some comments and add a bit of documentation.
Matti Hamalainen <ccr@tnsp.org>
parents:
49
diff
changeset
|
16 # if you wish to preserve the old data. |
| 0 | 17 # |
|
50
f69363fc1f61
Update some comments and add a bit of documentation.
Matti Hamalainen <ccr@tnsp.org>
parents:
49
diff
changeset
|
18 # See convert_urllog_db.tcl for more information. |
|
f69363fc1f61
Update some comments and add a bit of documentation.
Matti Hamalainen <ccr@tnsp.org>
parents:
49
diff
changeset
|
19 # |
|
81
17e542b7985a
urllog, quotedb: Improve documentation.
Matti Hamalainen <ccr@tnsp.org>
parents:
73
diff
changeset
|
20 # If you are doing a fresh install, you will need to create the |
|
50
f69363fc1f61
Update some comments and add a bit of documentation.
Matti Hamalainen <ccr@tnsp.org>
parents:
49
diff
changeset
|
21 # initial SQLite3 database with the required table schemas. You |
|
f69363fc1f61
Update some comments and add a bit of documentation.
Matti Hamalainen <ccr@tnsp.org>
parents:
49
diff
changeset
|
22 # can do that by running: create_urllog_db.tcl |
| 0 | 23 # |
| 24 ########################################################################## | |
|
13
e06d41fb69d5
Begin work on converting urllog.tcl to use an SQLite3 database instead of flat file.
Matti Hamalainen <ccr@tnsp.org>
parents:
8
diff
changeset
|
25 |
| 0 | 26 ### |
| 27 ### HTTP options | |
| 28 ### | |
|
81
17e542b7985a
urllog, quotedb: Improve documentation.
Matti Hamalainen <ccr@tnsp.org>
parents:
73
diff
changeset
|
29 # Set to 1 if you want to enable use of HTTP proxy. |
|
17e542b7985a
urllog, quotedb: Improve documentation.
Matti Hamalainen <ccr@tnsp.org>
parents:
73
diff
changeset
|
30 # If you do, you MUST set the proxy settings below too. |
| 0 | 31 set http_proxy 0 |
| 32 | |
| 33 # Proxy host and port number (only used if enabled above) | |
| 34 set http_proxy_host "" | |
| 35 set http_proxy_port 8080 | |
| 36 | |
|
81
17e542b7985a
urllog, quotedb: Improve documentation.
Matti Hamalainen <ccr@tnsp.org>
parents:
73
diff
changeset
|
37 # Enable _experimental_ TLS/SSL support. This may not work at all. |
|
17e542b7985a
urllog, quotedb: Improve documentation.
Matti Hamalainen <ccr@tnsp.org>
parents:
73
diff
changeset
|
38 # If unsure, leave this option disabled (0). |
|
104
da337ca10e0a
urllog: Enable SSL/TLS support by default.
Matti Hamalainen <ccr@tnsp.org>
parents:
103
diff
changeset
|
39 set http_tls_support 1 |
| 0 | 40 |
|
89
77e05ce9e9b8
urllog: Add certdir option setting.
Matti Hamalainen <ccr@tnsp.org>
parents:
87
diff
changeset
|
41 set http_tls_cadir "/usr/share/ca-certificates/mozilla" |
|
77e05ce9e9b8
urllog: Add certdir option setting.
Matti Hamalainen <ccr@tnsp.org>
parents:
87
diff
changeset
|
42 |
|
50
f69363fc1f61
Update some comments and add a bit of documentation.
Matti Hamalainen <ccr@tnsp.org>
parents:
49
diff
changeset
|
43 |
| 0 | 44 ### |
| 45 ### General options | |
| 46 ### | |
| 47 | |
|
219
4e09bcc48851
urllog: Add settings for specifying channels where URL logging is active, and where !urlfind functionality works (separately, if so desired.)
Matti Hamalainen <ccr@tnsp.org>
parents:
218
diff
changeset
|
48 # Channels where URLLog records links/URLs |
|
4e09bcc48851
urllog: Add settings for specifying channels where URL logging is active, and where !urlfind functionality works (separately, if so desired.)
Matti Hamalainen <ccr@tnsp.org>
parents:
218
diff
changeset
|
49 # set urllog_log_channels "#foobar;#baz" |
|
4e09bcc48851
urllog: Add settings for specifying channels where URL logging is active, and where !urlfind functionality works (separately, if so desired.)
Matti Hamalainen <ccr@tnsp.org>
parents:
218
diff
changeset
|
50 # You can use * to match substrings or everything |
|
4e09bcc48851
urllog: Add settings for specifying channels where URL logging is active, and where !urlfind functionality works (separately, if so desired.)
Matti Hamalainen <ccr@tnsp.org>
parents:
218
diff
changeset
|
51 set urllog_log_channels "*" |
|
4e09bcc48851
urllog: Add settings for specifying channels where URL logging is active, and where !urlfind functionality works (separately, if so desired.)
Matti Hamalainen <ccr@tnsp.org>
parents:
218
diff
changeset
|
52 |
|
4e09bcc48851
urllog: Add settings for specifying channels where URL logging is active, and where !urlfind functionality works (separately, if so desired.)
Matti Hamalainen <ccr@tnsp.org>
parents:
218
diff
changeset
|
53 |
|
4e09bcc48851
urllog: Add settings for specifying channels where URL logging is active, and where !urlfind functionality works (separately, if so desired.)
Matti Hamalainen <ccr@tnsp.org>
parents:
218
diff
changeset
|
54 |
|
50
f69363fc1f61
Update some comments and add a bit of documentation.
Matti Hamalainen <ccr@tnsp.org>
parents:
49
diff
changeset
|
55 # Filename of the SQLite URL database file |
|
13
e06d41fb69d5
Begin work on converting urllog.tcl to use an SQLite3 database instead of flat file.
Matti Hamalainen <ccr@tnsp.org>
parents:
8
diff
changeset
|
56 set urllog_db_file "urllog.sqlite" |
| 0 | 57 |
| 58 | |
| 59 # 1 = Verbose: Say messages when URL is OK, bad, etc. | |
| 60 # 0 = Quiet : Be quiet (only speak if asked with !urlfind, etc) | |
| 61 set urllog_verbose 1 | |
| 62 | |
| 63 | |
|
50
f69363fc1f61
Update some comments and add a bit of documentation.
Matti Hamalainen <ccr@tnsp.org>
parents:
49
diff
changeset
|
64 # 1 = Enable logging of various script actions into bot's log |
| 0 | 65 # 0 = Don't. |
| 66 set urllog_logmsg 1 | |
| 67 | |
| 68 | |
| 69 # 1 = Check URLs for validity and existence before adding. | |
| 70 # 0 = No checks. Add _anything_ that looks like an URL to the database. | |
| 71 set urllog_check 1 | |
| 72 | |
| 73 | |
| 74 ### | |
| 75 ### Search related settings | |
| 76 ### | |
| 77 | |
|
219
4e09bcc48851
urllog: Add settings for specifying channels where URL logging is active, and where !urlfind functionality works (separately, if so desired.)
Matti Hamalainen <ccr@tnsp.org>
parents:
218
diff
changeset
|
78 # Channels where !urlfind and other commands can be used. |
|
4e09bcc48851
urllog: Add settings for specifying channels where URL logging is active, and where !urlfind functionality works (separately, if so desired.)
Matti Hamalainen <ccr@tnsp.org>
parents:
218
diff
changeset
|
79 # By default this is set to be the same as urllog_log_channels |
|
4e09bcc48851
urllog: Add settings for specifying channels where URL logging is active, and where !urlfind functionality works (separately, if so desired.)
Matti Hamalainen <ccr@tnsp.org>
parents:
218
diff
changeset
|
80 set urllog_search_channels $urllog_log_channels |
| 0 | 81 |
|
81
17e542b7985a
urllog, quotedb: Improve documentation.
Matti Hamalainen <ccr@tnsp.org>
parents:
73
diff
changeset
|
82 # Limit how many URLs should the "!urlfind" command show at most. |
| 0 | 83 set urllog_showmax_pub 3 |
| 84 | |
|
81
17e542b7985a
urllog, quotedb: Improve documentation.
Matti Hamalainen <ccr@tnsp.org>
parents:
73
diff
changeset
|
85 # Same as above, but for private message search. |
| 0 | 86 set urllog_showmax_priv 6 |
| 87 | |
| 88 | |
| 89 ### | |
| 90 ### ShortURL-settings | |
| 91 ### | |
|
181
ff23ce8b938f
urllog: Document ShortURL functionality slightly.
Matti Hamalainen <ccr@tnsp.org>
parents:
176
diff
changeset
|
92 # To enable ShortURL functionality, you need to set up the |
|
ff23ce8b938f
urllog: Document ShortURL functionality slightly.
Matti Hamalainen <ccr@tnsp.org>
parents:
176
diff
changeset
|
93 # URL redirector PHP script (urlredirect.php) correctly, and |
|
ff23ce8b938f
urllog: Document ShortURL functionality slightly.
Matti Hamalainen <ccr@tnsp.org>
parents:
176
diff
changeset
|
94 # enable change the settings in it and below appropriately. |
|
ff23ce8b938f
urllog: Document ShortURL functionality slightly.
Matti Hamalainen <ccr@tnsp.org>
parents:
176
diff
changeset
|
95 # See urlredirect.php.txt for more information. |
|
ff23ce8b938f
urllog: Document ShortURL functionality slightly.
Matti Hamalainen <ccr@tnsp.org>
parents:
176
diff
changeset
|
96 # |
|
ff23ce8b938f
urllog: Document ShortURL functionality slightly.
Matti Hamalainen <ccr@tnsp.org>
parents:
176
diff
changeset
|
97 # You will also need SQLite3 support for PHP and access to |
|
ff23ce8b938f
urllog: Document ShortURL functionality slightly.
Matti Hamalainen <ccr@tnsp.org>
parents:
176
diff
changeset
|
98 # change .htaccess file(s) on your web server. The PHP |
|
ff23ce8b938f
urllog: Document ShortURL functionality slightly.
Matti Hamalainen <ccr@tnsp.org>
parents:
176
diff
changeset
|
99 # script will also need access to the SQLite3 database this |
|
ff23ce8b938f
urllog: Document ShortURL functionality slightly.
Matti Hamalainen <ccr@tnsp.org>
parents:
176
diff
changeset
|
100 # script uses. |
|
ff23ce8b938f
urllog: Document ShortURL functionality slightly.
Matti Hamalainen <ccr@tnsp.org>
parents:
176
diff
changeset
|
101 # |
| 0 | 102 |
|
73
646b2fd67312
urllog: Improve documentation of different settings.
Matti Hamalainen <ccr@tnsp.org>
parents:
70
diff
changeset
|
103 # 1 = Enable showing of ShortURLs |
|
646b2fd67312
urllog: Improve documentation of different settings.
Matti Hamalainen <ccr@tnsp.org>
parents:
70
diff
changeset
|
104 # 0 = ShortURLs not shown in any bot actions |
| 0 | 105 set urllog_shorturl 1 |
| 106 | |
|
73
646b2fd67312
urllog: Improve documentation of different settings.
Matti Hamalainen <ccr@tnsp.org>
parents:
70
diff
changeset
|
107 # Max length of original URL to be shown, rest is chopped |
|
646b2fd67312
urllog: Improve documentation of different settings.
Matti Hamalainen <ccr@tnsp.org>
parents:
70
diff
changeset
|
108 # off if the URL is longer than the specified amount. |
| 0 | 109 set urllog_shorturl_orig 30 |
| 110 | |
|
73
646b2fd67312
urllog: Improve documentation of different settings.
Matti Hamalainen <ccr@tnsp.org>
parents:
70
diff
changeset
|
111 # Web server URL that handles redirects of ShortURLs |
| 0 | 112 set urllog_shorturl_prefix "http://tnsp.org/u/" |
| 113 | |
| 114 | |
| 115 ### | |
|
81
17e542b7985a
urllog, quotedb: Improve documentation.
Matti Hamalainen <ccr@tnsp.org>
parents:
73
diff
changeset
|
116 ### Message texts (informal, errors, etc.) |
| 0 | 117 ### |
| 118 | |
| 119 # No such host was found | |
| 120 set urlmsg_nosuchhost "ei tommosta oo!" | |
| 121 | |
| 122 # Could not connect host (I/O errors etc) | |
| 123 set urlmsg_ioerror "kraak, virhe yhdynnässä." | |
| 124 | |
| 125 # HTTP timeout | |
| 126 set urlmsg_timeout "ei jaksa ootella" | |
| 127 | |
| 128 # No such document was found | |
| 129 set urlmsg_errorgettingdoc "siitosvirhe" | |
| 130 | |
| 131 # URL was already known (was in database) | |
| 132 set urlmsg_alreadyknown "wanha!" | |
| 133 #set urlmsg_alreadyknown "Empiiristen havaintojen perusteella ja tällä sovellutusalueella esiintyneisiin aikaisempiin kontekstuaalisiin ilmaisuihin viitaten uskallan todeta, että sovellukseen ilmoittamasi tietoverkko-osoite oli kronologisti ajatellen varsin postpresentuaalisesti sopimaton ja ennestään hyvin tunnettu." | |
| 134 | |
| 135 # No match was found when searched with !urlfind or other command | |
| 136 set urlmsg_nomatch "Ei osumia." | |
| 137 | |
| 138 | |
| 139 ### | |
| 140 ### Things that you usually don't need to touch ... | |
| 141 ### | |
| 142 | |
| 143 # What IRC "command" should we use to send messages: | |
| 144 # (Valid alternatives are "PRIVMSG" and "NOTICE") | |
| 145 set urllog_preferredmsg "PRIVMSG" | |
| 146 | |
| 147 # The valid known Top Level Domains (TLDs), but not the country code TLDs | |
| 148 # (Now includes the new IANA published TLDs) | |
|
90
a9a4456eb213
urllog: Add .xxx TLD to supported list.
Matti Hamalainen <ccr@tnsp.org>
parents:
89
diff
changeset
|
149 set urllog_tlds "org,com,net,mil,gov,biz,edu,coop,aero,info,museum,name,pro,int,xxx" |
| 0 | 150 |
| 151 | |
| 152 ########################################################################## | |
| 153 # No need to look below this line | |
| 154 ########################################################################## | |
| 155 set urllog_name "URLLog" | |
| 222 | 156 set urllog_version "2.3.0" |
| 0 | 157 |
| 158 set urllog_tlds [split $urllog_tlds ","] | |
| 159 set urllog_httprep [split "\@|%40|{|%7B|}|%7D|\[|%5B|\]|%5D" "|"] | |
| 160 | |
|
102
5425dc418505
urllog: Entity data is now in UTF-8, but TCL source files are interpreted with current system locale, which may not be UTF-8. We must therefore "convert" the entity mapping string to UTF-8 to be certain of TCL's interpretation of its encoding.
Matti Hamalainen <ccr@tnsp.org>
parents:
101
diff
changeset
|
161 |
|
131
b04ecf8bfb15
urllog: Fix some entity translations.
Matti Hamalainen <ccr@tnsp.org>
parents:
129
diff
changeset
|
162 set urllog_ent_str "-|-|'|'|—|-|‏||—|-|–|--|‪||‬|" |
|
b04ecf8bfb15
urllog: Fix some entity translations.
Matti Hamalainen <ccr@tnsp.org>
parents:
129
diff
changeset
|
163 append urllog_ent_str "|‎||å|Ã¥|Å|Ã…|é|é|:|:| | " |
| 133 | 164 append urllog_ent_str "|”|\"|“|\"|«|<<|»|>>|"|\"" |
|
131
b04ecf8bfb15
urllog: Fix some entity translations.
Matti Hamalainen <ccr@tnsp.org>
parents:
129
diff
changeset
|
165 append urllog_ent_str "|ä|ä|ö|ö|Ä|Ä|Ö|Ö|&|&|<|<|>|>" |
|
b04ecf8bfb15
urllog: Fix some entity translations.
Matti Hamalainen <ccr@tnsp.org>
parents:
129
diff
changeset
|
166 append urllog_ent_str "|ä|ä|å|ö|—|-|'|'|–|-|"|\"" |
|
b04ecf8bfb15
urllog: Fix some entity translations.
Matti Hamalainen <ccr@tnsp.org>
parents:
129
diff
changeset
|
167 append urllog_ent_str "|||-|’|'|ü|ü|Ü|Ãœ|•|*|€|€" |
|
223
606c2a48b2ce
urllog: Add one entity translation.
Matti Hamalainen <ccr@tnsp.org>
parents:
222
diff
changeset
|
168 append urllog_ent_str "|”|\"|‘|'" |
|
102
5425dc418505
urllog: Entity data is now in UTF-8, but TCL source files are interpreted with current system locale, which may not be UTF-8. We must therefore "convert" the entity mapping string to UTF-8 to be certain of TCL's interpretation of its encoding.
Matti Hamalainen <ccr@tnsp.org>
parents:
101
diff
changeset
|
169 set urllog_html_ent [split [encoding convertfrom "utf-8" $urllog_ent_str] "|"] |
| 0 | 170 |
|
13
e06d41fb69d5
Begin work on converting urllog.tcl to use an SQLite3 database instead of flat file.
Matti Hamalainen <ccr@tnsp.org>
parents:
8
diff
changeset
|
171 ### Require packages |
|
e06d41fb69d5
Begin work on converting urllog.tcl to use an SQLite3 database instead of flat file.
Matti Hamalainen <ccr@tnsp.org>
parents:
8
diff
changeset
|
172 package require sqlite3 |
| 0 | 173 package require http |
|
7
50b52294e93e
urllog: Strip ‏ entities from titles; Some work on SSL/https support.
Matti Hamalainen <ccr@tnsp.org>
parents:
4
diff
changeset
|
174 |
| 0 | 175 ### Binding initializations |
|
219
4e09bcc48851
urllog: Add settings for specifying channels where URL logging is active, and where !urlfind functionality works (separately, if so desired.)
Matti Hamalainen <ccr@tnsp.org>
parents:
218
diff
changeset
|
176 bind pub - !urlfind urllog_pub_urlfind |
|
4e09bcc48851
urllog: Add settings for specifying channels where URL logging is active, and where !urlfind functionality works (separately, if so desired.)
Matti Hamalainen <ccr@tnsp.org>
parents:
218
diff
changeset
|
177 bind msg - !urlfind urllog_msg_urlfind |
| 0 | 178 bind pubm - *.* urllog_checkmsg |
| 179 bind topc - *.* urllog_checkmsg | |
| 180 | |
| 181 | |
| 182 ### Initialization messages | |
|
176
eda776bcb7ed
urllog: Bump copyright, and version.
Matti Hamalainen <ccr@tnsp.org>
parents:
160
diff
changeset
|
183 set urllog_message "$urllog_name v$urllog_version (C) 2000-2014 ccr/TNSP" |
| 0 | 184 putlog "$urllog_message" |
| 185 | |
|
13
e06d41fb69d5
Begin work on converting urllog.tcl to use an SQLite3 database instead of flat file.
Matti Hamalainen <ccr@tnsp.org>
parents:
8
diff
changeset
|
186 ### HTTP module initialization |
|
e06d41fb69d5
Begin work on converting urllog.tcl to use an SQLite3 database instead of flat file.
Matti Hamalainen <ccr@tnsp.org>
parents:
8
diff
changeset
|
187 ::http::config -useragent "$urllog_name/$urllog_version" |
|
e06d41fb69d5
Begin work on converting urllog.tcl to use an SQLite3 database instead of flat file.
Matti Hamalainen <ccr@tnsp.org>
parents:
8
diff
changeset
|
188 if {$http_proxy != 0} { |
| 28 | 189 ::http::config -proxyhost $http_proxy_host -proxyport $http_proxy_port |
|
13
e06d41fb69d5
Begin work on converting urllog.tcl to use an SQLite3 database instead of flat file.
Matti Hamalainen <ccr@tnsp.org>
parents:
8
diff
changeset
|
190 } |
|
e06d41fb69d5
Begin work on converting urllog.tcl to use an SQLite3 database instead of flat file.
Matti Hamalainen <ccr@tnsp.org>
parents:
8
diff
changeset
|
191 |
|
e06d41fb69d5
Begin work on converting urllog.tcl to use an SQLite3 database instead of flat file.
Matti Hamalainen <ccr@tnsp.org>
parents:
8
diff
changeset
|
192 if {$http_tls_support != 0} { |
| 28 | 193 package require tls |
|
89
77e05ce9e9b8
urllog: Add certdir option setting.
Matti Hamalainen <ccr@tnsp.org>
parents:
87
diff
changeset
|
194 ::http::register https 443 [list ::tls::socket -request 1 -require 1 -cadir $http_tls_cadir] |
|
13
e06d41fb69d5
Begin work on converting urllog.tcl to use an SQLite3 database instead of flat file.
Matti Hamalainen <ccr@tnsp.org>
parents:
8
diff
changeset
|
195 } |
|
e06d41fb69d5
Begin work on converting urllog.tcl to use an SQLite3 database instead of flat file.
Matti Hamalainen <ccr@tnsp.org>
parents:
8
diff
changeset
|
196 |
|
e06d41fb69d5
Begin work on converting urllog.tcl to use an SQLite3 database instead of flat file.
Matti Hamalainen <ccr@tnsp.org>
parents:
8
diff
changeset
|
197 ### SQLite database initialization |
|
e06d41fb69d5
Begin work on converting urllog.tcl to use an SQLite3 database instead of flat file.
Matti Hamalainen <ccr@tnsp.org>
parents:
8
diff
changeset
|
198 if {[catch {sqlite3 urldb $urllog_db_file} uerrmsg]} { |
| 28 | 199 putlog " Could not open SQLite3 database '$urllog_db_file': $uerrmsg" |
| 200 exit 2 | |
|
13
e06d41fb69d5
Begin work on converting urllog.tcl to use an SQLite3 database instead of flat file.
Matti Hamalainen <ccr@tnsp.org>
parents:
8
diff
changeset
|
201 } |
|
e06d41fb69d5
Begin work on converting urllog.tcl to use an SQLite3 database instead of flat file.
Matti Hamalainen <ccr@tnsp.org>
parents:
8
diff
changeset
|
202 |
|
e06d41fb69d5
Begin work on converting urllog.tcl to use an SQLite3 database instead of flat file.
Matti Hamalainen <ccr@tnsp.org>
parents:
8
diff
changeset
|
203 |
| 0 | 204 if {$http_proxy != 0} { |
| 28 | 205 putlog " (Using proxy $http_proxy_host:$http_proxy_port)" |
| 0 | 206 } |
| 207 | |
| 208 if {$urllog_check != 0} { | |
| 28 | 209 putlog " (Additional URL validity checks enabled)" |
| 0 | 210 } |
| 211 | |
| 212 if {$urllog_verbose != 0} { | |
| 28 | 213 putlog " (Verbose mode enabled)" |
| 0 | 214 } |
| 215 | |
| 216 #------------------------------------------------------------------------- | |
| 217 ### Utility functions | |
| 218 proc urllog_log {arg} { | |
| 28 | 219 global urllog_logmsg urllog_name |
| 0 | 220 |
| 28 | 221 if {$urllog_logmsg != 0} { |
| 222 putlog "$urllog_name: $arg" | |
| 223 } | |
| 0 | 224 } |
| 225 | |
| 226 | |
| 152 | 227 proc urllog_ctime {utime} { |
| 28 | 228 if {$utime == "" || $utime == "*"} { |
| 229 set utime 0 | |
| 230 } | |
| 231 return [clock format $utime -format "%d.%m.%Y %H:%M"] | |
| 0 | 232 } |
| 233 | |
| 234 | |
| 235 proc urllog_isnumber {uarg} { | |
| 28 | 236 foreach i [split $uarg {}] { |
|
65
31c8c4f50aa6
urllog: Improve urllog_isnumber function.
Matti Hamalainen <ccr@tnsp.org>
parents:
62
diff
changeset
|
237 if {![string match \[0-9\] $i]} { return 0 } |
| 28 | 238 } |
|
65
31c8c4f50aa6
urllog: Improve urllog_isnumber function.
Matti Hamalainen <ccr@tnsp.org>
parents:
62
diff
changeset
|
239 return 1 |
| 0 | 240 } |
| 241 | |
| 242 | |
| 243 proc urllog_msg {apublic anick achan amsg} { | |
| 28 | 244 global urllog_preferredmsg |
| 0 | 245 |
| 28 | 246 if {$apublic == 1} { |
| 247 putserv "$urllog_preferredmsg $achan :$amsg" | |
| 248 } else { | |
| 249 putserv "$urllog_preferredmsg $anick :$amsg" | |
| 250 } | |
| 0 | 251 } |
| 252 | |
| 253 | |
| 254 proc urllog_verb_msg {anick achan amsg} { | |
| 28 | 255 global urllog_verbose |
| 0 | 256 |
| 28 | 257 if {$urllog_verbose != 0} { |
| 258 urllog_msg 1 $anick $achan $amsg | |
| 259 } | |
| 0 | 260 } |
| 261 | |
| 262 | |
| 263 proc urllog_convert_ent {udata} { | |
| 28 | 264 global urllog_html_ent |
|
115
5db02af76016
urllog: Improve entity conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
114
diff
changeset
|
265 return [string map -nocase $urllog_html_ent [string map $urllog_html_ent $udata]] |
| 0 | 266 } |
| 267 | |
| 268 | |
|
13
e06d41fb69d5
Begin work on converting urllog.tcl to use an SQLite3 database instead of flat file.
Matti Hamalainen <ccr@tnsp.org>
parents:
8
diff
changeset
|
269 proc urllog_escape { str } { |
| 28 | 270 return [string map {' ''} $str] |
|
13
e06d41fb69d5
Begin work on converting urllog.tcl to use an SQLite3 database instead of flat file.
Matti Hamalainen <ccr@tnsp.org>
parents:
8
diff
changeset
|
271 } |
|
e06d41fb69d5
Begin work on converting urllog.tcl to use an SQLite3 database instead of flat file.
Matti Hamalainen <ccr@tnsp.org>
parents:
8
diff
changeset
|
272 |
|
116
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
273 |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
274 proc urllog_sanitize_encoding {uencoding} { |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
275 regsub -- "^\[a-z\]\[a-z\]_\[A-Z\]\[A-Z\]\." $uencoding "" uencoding |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
276 set uencoding [string tolower $uencoding] |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
277 regsub -- "^iso-" $uencoding "iso" uencoding |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
278 return $uencoding |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
279 } |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
280 |
|
121
bec98a9f8695
Convert the title encoding when outputting to channel.
Matti Hamalainen <ccr@tnsp.org>
parents:
120
diff
changeset
|
281 proc urllog_clean_title {utitle} { |
|
bec98a9f8695
Convert the title encoding when outputting to channel.
Matti Hamalainen <ccr@tnsp.org>
parents:
120
diff
changeset
|
282 if {[catch {set utitle [encoding convertto "iso8859-15" $utitle]} cerrmsg]} { |
|
bec98a9f8695
Convert the title encoding when outputting to channel.
Matti Hamalainen <ccr@tnsp.org>
parents:
120
diff
changeset
|
283 putlog "Could not convert title encoding: $cerrmsg" |
|
bec98a9f8695
Convert the title encoding when outputting to channel.
Matti Hamalainen <ccr@tnsp.org>
parents:
120
diff
changeset
|
284 } |
|
bec98a9f8695
Convert the title encoding when outputting to channel.
Matti Hamalainen <ccr@tnsp.org>
parents:
120
diff
changeset
|
285 return $utitle |
|
bec98a9f8695
Convert the title encoding when outputting to channel.
Matti Hamalainen <ccr@tnsp.org>
parents:
120
diff
changeset
|
286 } |
|
116
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
287 |
| 0 | 288 #------------------------------------------------------------------------- |
|
150
52350ed97775
urllog: Cleanups, rename/move some global variables.
Matti Hamalainen <ccr@tnsp.org>
parents:
136
diff
changeset
|
289 set urllog_shorturl_str "ABCDEFGHIJKLNMOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789" |
|
13
e06d41fb69d5
Begin work on converting urllog.tcl to use an SQLite3 database instead of flat file.
Matti Hamalainen <ccr@tnsp.org>
parents:
8
diff
changeset
|
290 |
|
150
52350ed97775
urllog: Cleanups, rename/move some global variables.
Matti Hamalainen <ccr@tnsp.org>
parents:
136
diff
changeset
|
291 proc urllog_get_short {utime} { |
|
52350ed97775
urllog: Cleanups, rename/move some global variables.
Matti Hamalainen <ccr@tnsp.org>
parents:
136
diff
changeset
|
292 global urllog_shorturl_prefix urllog_shorturl_str |
|
52350ed97775
urllog: Cleanups, rename/move some global variables.
Matti Hamalainen <ccr@tnsp.org>
parents:
136
diff
changeset
|
293 |
|
52350ed97775
urllog: Cleanups, rename/move some global variables.
Matti Hamalainen <ccr@tnsp.org>
parents:
136
diff
changeset
|
294 set ulen [string length $urllog_shorturl_str] |
| 0 | 295 |
| 28 | 296 set u1 [expr $utime / ($ulen * $ulen)] |
| 297 set utmp [expr $utime % ($ulen * $ulen)] | |
| 298 set u2 [expr $utmp / $ulen] | |
| 299 set u3 [expr $utmp % $ulen] | |
| 0 | 300 |
|
150
52350ed97775
urllog: Cleanups, rename/move some global variables.
Matti Hamalainen <ccr@tnsp.org>
parents:
136
diff
changeset
|
301 return "\[ $urllog_shorturl_prefix[string index $urllog_shorturl_str $u1][string index $urllog_shorturl_str $u2][string index $urllog_shorturl_str $u3] \]" |
|
52350ed97775
urllog: Cleanups, rename/move some global variables.
Matti Hamalainen <ccr@tnsp.org>
parents:
136
diff
changeset
|
302 } |
| 0 | 303 } |
| 304 | |
| 305 | |
| 306 #------------------------------------------------------------------------- | |
| 307 proc urllog_chop_url {url} { | |
| 28 | 308 global urllog_shorturl_orig |
| 68 | 309 |
| 28 | 310 if {[string length $url] > $urllog_shorturl_orig} { |
| 311 return "[string range $url 0 $urllog_shorturl_orig]..." | |
| 312 } else { | |
| 313 return $url | |
| 314 } | |
| 0 | 315 } |
| 316 | |
| 317 #------------------------------------------------------------------------- | |
|
83
f171a9fb7b7b
urllog: Split urllog_add function to urllog_exists for checking whether given URL already exists in the database. Use urllog_exists where appropriate.
Matti Hamalainen <ccr@tnsp.org>
parents:
82
diff
changeset
|
318 proc urllog_exists {urlStr urlNick urlHost urlChan} { |
| 28 | 319 global urldb urlmsg_alreadyknown urllog_shorturl |
| 0 | 320 |
|
83
f171a9fb7b7b
urllog: Split urllog_add function to urllog_exists for checking whether given URL already exists in the database. Use urllog_exists where appropriate.
Matti Hamalainen <ccr@tnsp.org>
parents:
82
diff
changeset
|
321 set usql "SELECT id AS uid, utime AS utime, url AS uurl, user AS uuser, host AS uhost, chan AS uchan, title AS utitle FROM urls WHERE url='[urllog_escape $urlStr]'" |
|
f171a9fb7b7b
urllog: Split urllog_add function to urllog_exists for checking whether given URL already exists in the database. Use urllog_exists where appropriate.
Matti Hamalainen <ccr@tnsp.org>
parents:
82
diff
changeset
|
322 urldb eval $usql { |
| 28 | 323 urllog_log "URL said by $urlNick ($urlStr) already known" |
| 324 if {$urllog_shorturl != 0} { | |
|
83
f171a9fb7b7b
urllog: Split urllog_add function to urllog_exists for checking whether given URL already exists in the database. Use urllog_exists where appropriate.
Matti Hamalainen <ccr@tnsp.org>
parents:
82
diff
changeset
|
325 set qstr "[urllog_get_short $uid] " |
| 28 | 326 } else { |
| 327 set qstr "" | |
| 328 } | |
| 329 append qstr "($uuser/$uchan@[urllog_ctime $utime])" | |
|
83
f171a9fb7b7b
urllog: Split urllog_add function to urllog_exists for checking whether given URL already exists in the database. Use urllog_exists where appropriate.
Matti Hamalainen <ccr@tnsp.org>
parents:
82
diff
changeset
|
330 if {[string length $utitle] > 0} { |
|
121
bec98a9f8695
Convert the title encoding when outputting to channel.
Matti Hamalainen <ccr@tnsp.org>
parents:
120
diff
changeset
|
331 set qstr "$urlmsg_alreadyknown - '[urllog_clean_title $utitle]' $qstr" |
| 28 | 332 } else { |
| 333 set qstr "$urlmsg_alreadyknown $qstr" | |
| 334 } | |
| 335 urllog_verb_msg $urlNick $urlChan $qstr | |
| 336 return 0 | |
| 337 } | |
|
83
f171a9fb7b7b
urllog: Split urllog_add function to urllog_exists for checking whether given URL already exists in the database. Use urllog_exists where appropriate.
Matti Hamalainen <ccr@tnsp.org>
parents:
82
diff
changeset
|
338 return 1 |
|
f171a9fb7b7b
urllog: Split urllog_add function to urllog_exists for checking whether given URL already exists in the database. Use urllog_exists where appropriate.
Matti Hamalainen <ccr@tnsp.org>
parents:
82
diff
changeset
|
339 } |
| 0 | 340 |
|
18
1e2232135354
More changes for SQLite support.
Matti Hamalainen <ccr@tnsp.org>
parents:
13
diff
changeset
|
341 |
|
83
f171a9fb7b7b
urllog: Split urllog_add function to urllog_exists for checking whether given URL already exists in the database. Use urllog_exists where appropriate.
Matti Hamalainen <ccr@tnsp.org>
parents:
82
diff
changeset
|
342 #------------------------------------------------------------------------- |
|
f171a9fb7b7b
urllog: Split urllog_add function to urllog_exists for checking whether given URL already exists in the database. Use urllog_exists where appropriate.
Matti Hamalainen <ccr@tnsp.org>
parents:
82
diff
changeset
|
343 proc urllog_addurl {urlStr urlNick urlHost urlChan urlTitle} { |
|
f171a9fb7b7b
urllog: Split urllog_add function to urllog_exists for checking whether given URL already exists in the database. Use urllog_exists where appropriate.
Matti Hamalainen <ccr@tnsp.org>
parents:
82
diff
changeset
|
344 global urldb urllog_shorturl |
|
f171a9fb7b7b
urllog: Split urllog_add function to urllog_exists for checking whether given URL already exists in the database. Use urllog_exists where appropriate.
Matti Hamalainen <ccr@tnsp.org>
parents:
82
diff
changeset
|
345 |
|
93
4e02c0219afe
urllog: Insert NULL into title column when we didn't get a title.
Matti Hamalainen <ccr@tnsp.org>
parents:
92
diff
changeset
|
346 if {$urlTitle == ""} { |
|
4e02c0219afe
urllog: Insert NULL into title column when we didn't get a title.
Matti Hamalainen <ccr@tnsp.org>
parents:
92
diff
changeset
|
347 set uins "NULL" |
|
4e02c0219afe
urllog: Insert NULL into title column when we didn't get a title.
Matti Hamalainen <ccr@tnsp.org>
parents:
92
diff
changeset
|
348 } else { |
|
4e02c0219afe
urllog: Insert NULL into title column when we didn't get a title.
Matti Hamalainen <ccr@tnsp.org>
parents:
92
diff
changeset
|
349 set uins "'[urllog_escape $urlTitle]'" |
|
4e02c0219afe
urllog: Insert NULL into title column when we didn't get a title.
Matti Hamalainen <ccr@tnsp.org>
parents:
92
diff
changeset
|
350 } |
|
4e02c0219afe
urllog: Insert NULL into title column when we didn't get a title.
Matti Hamalainen <ccr@tnsp.org>
parents:
92
diff
changeset
|
351 set usql "INSERT INTO urls (utime,url,user,host,chan,title) VALUES ([unixtime], '[urllog_escape $urlStr]', '[urllog_escape $urlNick]', '[urllog_escape $urlHost]', '[urllog_escape $urlChan]', $uins)" |
|
83
f171a9fb7b7b
urllog: Split urllog_add function to urllog_exists for checking whether given URL already exists in the database. Use urllog_exists where appropriate.
Matti Hamalainen <ccr@tnsp.org>
parents:
82
diff
changeset
|
352 if {[catch {urldb eval $usql} uerrmsg]} { |
|
f171a9fb7b7b
urllog: Split urllog_add function to urllog_exists for checking whether given URL already exists in the database. Use urllog_exists where appropriate.
Matti Hamalainen <ccr@tnsp.org>
parents:
82
diff
changeset
|
353 urllog_log "$uerrmsg on SQL:\n$usql" |
| 28 | 354 return 0 |
| 355 } | |
|
82
1bbc79f41a1c
urllog: Rename few variables for clarity.
Matti Hamalainen <ccr@tnsp.org>
parents:
81
diff
changeset
|
356 set uid [urldb last_insert_rowid] |
| 28 | 357 urllog_log "Added URL ($urlNick@$urlChan): $urlStr" |
| 0 | 358 |
| 359 | |
| 28 | 360 ### Let's say something, to confirm that everything went well. |
| 361 if {$urllog_shorturl != 0} { | |
|
82
1bbc79f41a1c
urllog: Rename few variables for clarity.
Matti Hamalainen <ccr@tnsp.org>
parents:
81
diff
changeset
|
362 set qstr "[urllog_get_short $uid] " |
| 28 | 363 } else { |
| 364 set qstr "" | |
| 365 } | |
| 366 if {[string length $urlTitle] > 0} { | |
|
121
bec98a9f8695
Convert the title encoding when outputting to channel.
Matti Hamalainen <ccr@tnsp.org>
parents:
120
diff
changeset
|
367 urllog_verb_msg $urlNick $urlChan "'[urllog_clean_title $urlTitle]' ([urllog_chop_url $urlStr]) $qstr" |
| 28 | 368 } else { |
| 369 urllog_verb_msg $urlNick $urlChan "[urllog_chop_url $urlStr] $qstr" | |
| 370 } | |
| 0 | 371 |
| 28 | 372 return 1 |
| 0 | 373 } |
| 374 | |
| 375 | |
| 376 #------------------------------------------------------------------------- | |
|
3
8003090caa35
Lots of code cleanups, add "fixer" for RasiaTube links (which suck) to point directly to Youtube.
Matti Hamalainen <ccr@tnsp.org>
parents:
0
diff
changeset
|
377 proc urllog_http_handler {utoken utotal ucurr} { |
| 28 | 378 upvar #0 $utoken state |
| 0 | 379 |
| 28 | 380 # Stop fetching data after 3000 bytes, this should be enough to |
| 381 # contain the head section of a HTML page. | |
| 382 if {$ucurr > 64000} { | |
| 383 set state(status) "ok" | |
| 384 } | |
| 0 | 385 } |
| 386 | |
|
84
fa1e95c2a0bc
urllog: Bump version to 2.1.
Matti Hamalainen <ccr@tnsp.org>
parents:
83
diff
changeset
|
387 |
| 0 | 388 #------------------------------------------------------------------------- |
| 389 proc urllog_checkurl {urlStr urlNick urlHost urlChan} { | |
| 28 | 390 global urllog_tlds urllog_check urlmsg_nosuchhost urlmsg_ioerror |
| 391 global urlmsg_timeout urlmsg_errorgettingdoc urllog_httprep | |
| 392 global urllog_shorturl_prefix urllog_shorturl urllog_encoding | |
|
95
687bdd74dfac
urllog: Check if TLS support is enabled when checking if we can fetch title information via HTTP or SSL/HTTP.
Matti Hamalainen <ccr@tnsp.org>
parents:
93
diff
changeset
|
393 global http_tls_support |
|
3
8003090caa35
Lots of code cleanups, add "fixer" for RasiaTube links (which suck) to point directly to Youtube.
Matti Hamalainen <ccr@tnsp.org>
parents:
0
diff
changeset
|
394 |
|
96
e5a6c27be365
urllog: Comments and cosmetics.
Matti Hamalainen <ccr@tnsp.org>
parents:
95
diff
changeset
|
395 ### Try to guess the URL protocol component (if it is missing) |
| 28 | 396 set u_checktld 1 |
| 397 if {[string match "*www.*" $urlStr] && ![string match "http://*" $urlStr] && ![string match "https://*" $urlStr]} { | |
| 398 set urlStr "http://$urlStr" | |
| 399 } elseif {[string match "*ftp.*" $urlStr] && ![string match "ftp://*" $urlStr]} { | |
| 400 set urlStr "ftp://$urlStr" | |
| 401 } | |
| 0 | 402 |
|
95
687bdd74dfac
urllog: Check if TLS support is enabled when checking if we can fetch title information via HTTP or SSL/HTTP.
Matti Hamalainen <ccr@tnsp.org>
parents:
93
diff
changeset
|
403 ### Handle URLs that have an IPv4-address |
|
687bdd74dfac
urllog: Check if TLS support is enabled when checking if we can fetch title information via HTTP or SSL/HTTP.
Matti Hamalainen <ccr@tnsp.org>
parents:
93
diff
changeset
|
404 if {[regexp "(\[a-z\]+)://(\[0-9\]{1,3})\\.(\[0-9\]{1,3})\\.(\[0-9\]{1,3})\\.(\[0-9\]{1,3})" $urlStr u_match u_proto ni1 ni2 ni3 ni4]} { |
| 28 | 405 # Check if the IP is on local network |
|
92
f6f4595856ff
urllog: Cosmetics. Remove useless parenthesis.
Matti Hamalainen <ccr@tnsp.org>
parents:
91
diff
changeset
|
406 if {$ni1 == 127 || $ni1 == 10 || ($ni1 == 192 && $ni2 == 168) || $ni1 == 0} { |
| 28 | 407 urllog_log "URL pointing to local or invalid network, ignored ($urlStr)." |
| 408 return 0 | |
| 409 } | |
| 410 # Skip TLD check for URLs with IP address | |
| 411 set u_checktld 0 | |
| 412 } | |
| 0 | 413 |
|
96
e5a6c27be365
urllog: Comments and cosmetics.
Matti Hamalainen <ccr@tnsp.org>
parents:
95
diff
changeset
|
414 ### Check now if we have an ShortURL here ... |
|
150
52350ed97775
urllog: Cleanups, rename/move some global variables.
Matti Hamalainen <ccr@tnsp.org>
parents:
136
diff
changeset
|
415 if {[string match "$urllog_shorturl_prefix*" $urlStr]} { |
|
98
fbbe7ee40e2f
urllog: Improve one informational / error message.
Matti Hamalainen <ccr@tnsp.org>
parents:
97
diff
changeset
|
416 urllog_log "Ignoring ShortURL from $urlNick: $urlStr" |
|
150
52350ed97775
urllog: Cleanups, rename/move some global variables.
Matti Hamalainen <ccr@tnsp.org>
parents:
136
diff
changeset
|
417 set uud "" |
|
52350ed97775
urllog: Cleanups, rename/move some global variables.
Matti Hamalainen <ccr@tnsp.org>
parents:
136
diff
changeset
|
418 set usql "SELECT id AS uid, url AS uurl, user AS uuser, host AS uhost, chan AS uchan, title AS utitle FROM urls WHERE utime=$uud" |
|
52350ed97775
urllog: Cleanups, rename/move some global variables.
Matti Hamalainen <ccr@tnsp.org>
parents:
136
diff
changeset
|
419 urldb eval $usql { |
|
52350ed97775
urllog: Cleanups, rename/move some global variables.
Matti Hamalainen <ccr@tnsp.org>
parents:
136
diff
changeset
|
420 |
|
52350ed97775
urllog: Cleanups, rename/move some global variables.
Matti Hamalainen <ccr@tnsp.org>
parents:
136
diff
changeset
|
421 } |
| 28 | 422 return 0 |
| 423 } | |
| 0 | 424 |
|
95
687bdd74dfac
urllog: Check if TLS support is enabled when checking if we can fetch title information via HTTP or SSL/HTTP.
Matti Hamalainen <ccr@tnsp.org>
parents:
93
diff
changeset
|
425 ### Get URL protocol component |
|
687bdd74dfac
urllog: Check if TLS support is enabled when checking if we can fetch title information via HTTP or SSL/HTTP.
Matti Hamalainen <ccr@tnsp.org>
parents:
93
diff
changeset
|
426 set u_proto "" |
|
160
e3e156911ab4
urllog: Oops, fix a silly typobug.
Matti Hamalainen <ccr@tnsp.org>
parents:
155
diff
changeset
|
427 regexp "(\[a-z\]+)://" $urlStr u_match u_proto |
|
95
687bdd74dfac
urllog: Check if TLS support is enabled when checking if we can fetch title information via HTTP or SSL/HTTP.
Matti Hamalainen <ccr@tnsp.org>
parents:
93
diff
changeset
|
428 |
| 28 | 429 ### Check the PORT (if the ":" is there) |
| 430 set u_record [split $urlStr "/"] | |
| 431 set u_hostname [lindex $u_record 2] | |
| 432 set u_port [lindex [split $u_hostname ":"] end] | |
| 0 | 433 |
| 28 | 434 if {![urllog_isnumber $u_port] && $u_port != "" && $u_port != $u_hostname} { |
| 435 urllog_log "Broken URL from $urlNick: ($urlStr) illegal port $u_port" | |
| 436 return 0 | |
| 437 } | |
| 0 | 438 |
| 28 | 439 # Default to port 80 (HTTP) |
| 440 if {![urllog_isnumber $u_port]} { | |
| 441 set u_port 80 | |
| 442 } | |
|
3
8003090caa35
Lots of code cleanups, add "fixer" for RasiaTube links (which suck) to point directly to Youtube.
Matti Hamalainen <ccr@tnsp.org>
parents:
0
diff
changeset
|
443 |
| 28 | 444 ### Is it a http or ftp url? (FIX ME!) |
|
97
366e68ad94df
urllog: Use u_proto variable to check for if the protocol is supported instead of doing useless additional string checking.
Matti Hamalainen <ccr@tnsp.org>
parents:
96
diff
changeset
|
445 if {$u_proto != "http" && $u_proto != "https" && $u_proto != "ftp"} { |
|
366e68ad94df
urllog: Use u_proto variable to check for if the protocol is supported instead of doing useless additional string checking.
Matti Hamalainen <ccr@tnsp.org>
parents:
96
diff
changeset
|
446 urllog_log "Broken URL from $urlNick: ($urlStr) UNSUPPORTED protocol class ($u_proto)." |
| 28 | 447 return 0 |
| 448 } | |
| 0 | 449 |
| 28 | 450 ### Check the Top Level Domain (TLD) validity |
| 451 if {$u_checktld != 0} { | |
| 452 set u_sane [lindex [split $u_hostname "."] end] | |
| 453 set u_tld [lindex [split $u_sane ":"] 0] | |
| 454 set u_found 0 | |
| 0 | 455 |
| 28 | 456 if {[string length $u_tld] == 2} { |
| 457 # Assume all 2-letter domains to be valid :) | |
| 458 set u_found 1 | |
| 459 } else { | |
| 460 # Check our list of known TLDs | |
| 461 foreach itld $urllog_tlds { | |
| 462 if {[string match $itld $u_tld]} { | |
| 463 set u_found 1 | |
| 464 } | |
| 465 } | |
| 466 } | |
| 0 | 467 |
| 28 | 468 if {$u_found == 0} { |
| 469 urllog_log "Broken URL from $urlNick: ($urlStr) illegal TLD: $u_tld." | |
| 470 return 0 | |
| 471 } | |
| 472 } | |
| 0 | 473 |
| 28 | 474 set urlStr [string map $urllog_httprep $urlStr] |
|
3
8003090caa35
Lots of code cleanups, add "fixer" for RasiaTube links (which suck) to point directly to Youtube.
Matti Hamalainen <ccr@tnsp.org>
parents:
0
diff
changeset
|
475 |
|
91
6f4bfd8e9447
urllog: Reorder code and make it simpler by removing duplicate checks.
Matti Hamalainen <ccr@tnsp.org>
parents:
90
diff
changeset
|
476 ### Does the URL already exist? |
|
6f4bfd8e9447
urllog: Reorder code and make it simpler by removing duplicate checks.
Matti Hamalainen <ccr@tnsp.org>
parents:
90
diff
changeset
|
477 if {![urllog_exists $urlStr $urlNick $urlHost $urlChan]} { |
|
6f4bfd8e9447
urllog: Reorder code and make it simpler by removing duplicate checks.
Matti Hamalainen <ccr@tnsp.org>
parents:
90
diff
changeset
|
478 return 1 |
|
6f4bfd8e9447
urllog: Reorder code and make it simpler by removing duplicate checks.
Matti Hamalainen <ccr@tnsp.org>
parents:
90
diff
changeset
|
479 } |
| 0 | 480 |
| 28 | 481 ### Do we perform additional optional checks? |
|
136
76eefceb2b90
Fix additional checks for HTTPS urls.
Matti Hamalainen <ccr@tnsp.org>
parents:
133
diff
changeset
|
482 if {$urllog_check != 0 && (($http_tls_support != 0 && $u_proto == "https") || $u_proto == "http")} { |
|
76eefceb2b90
Fix additional checks for HTTPS urls.
Matti Hamalainen <ccr@tnsp.org>
parents:
133
diff
changeset
|
483 # Ok |
|
76eefceb2b90
Fix additional checks for HTTPS urls.
Matti Hamalainen <ccr@tnsp.org>
parents:
133
diff
changeset
|
484 } else { |
|
83
f171a9fb7b7b
urllog: Split urllog_add function to urllog_exists for checking whether given URL already exists in the database. Use urllog_exists where appropriate.
Matti Hamalainen <ccr@tnsp.org>
parents:
82
diff
changeset
|
485 # No optional checks, just add the URL, if it does not exist already |
|
91
6f4bfd8e9447
urllog: Reorder code and make it simpler by removing duplicate checks.
Matti Hamalainen <ccr@tnsp.org>
parents:
90
diff
changeset
|
486 urllog_addurl $urlStr $urlNick $urlHost $urlChan "" |
| 28 | 487 return 1 |
| 488 } | |
|
7
50b52294e93e
urllog: Strip ‏ entities from titles; Some work on SSL/https support.
Matti Hamalainen <ccr@tnsp.org>
parents:
4
diff
changeset
|
489 |
| 28 | 490 ### Does the document pointed by the URL exist? |
|
209
6b1a579fb288
urllog: Accept only uncompressed data, because we use chunked download.
Matti Hamalainen <ccr@tnsp.org>
parents:
205
diff
changeset
|
491 if {[catch {set utoken [::http::geturl $urlStr -progress urllog_http_handler -blocksize 1024 -timeout 6000 -headers {Accept-Encoding identity}]} uerrmsg]} { |
| 28 | 492 urllog_verb_msg $urlNick $urlChan "$urlmsg_ioerror ($uerrmsg)" |
| 493 urllog_log "HTTP request failed: $uerrmsg" | |
| 494 return 0 | |
| 495 } | |
| 0 | 496 |
|
224
aaf433ab696a
urllog: Improve error messages a bit.
Matti Hamalainen <ccr@tnsp.org>
parents:
223
diff
changeset
|
497 set ustatus [::http::status $utoken] |
|
aaf433ab696a
urllog: Improve error messages a bit.
Matti Hamalainen <ccr@tnsp.org>
parents:
223
diff
changeset
|
498 if {$ustatus == "timeout"} { |
| 28 | 499 urllog_verb_msg $urlNick $urlChan "$urlmsg_timeout" |
| 500 urllog_log "HTTP request timed out ($urlStr)" | |
| 501 return 0 | |
| 502 } | |
| 0 | 503 |
|
224
aaf433ab696a
urllog: Improve error messages a bit.
Matti Hamalainen <ccr@tnsp.org>
parents:
223
diff
changeset
|
504 if {$ustatus != "ok"} { |
| 28 | 505 urllog_verb_msg $urlNick $urlChan "$urlmsg_errorgettingdoc ([::http::error $utoken])" |
| 506 urllog_log "Error in HTTP transaction: [::http::error $utoken] ($urlStr)" | |
| 507 return 0 | |
| 508 } | |
|
3
8003090caa35
Lots of code cleanups, add "fixer" for RasiaTube links (which suck) to point directly to Youtube.
Matti Hamalainen <ccr@tnsp.org>
parents:
0
diff
changeset
|
509 |
| 28 | 510 # Fixme! Handle redirects! |
|
224
aaf433ab696a
urllog: Improve error messages a bit.
Matti Hamalainen <ccr@tnsp.org>
parents:
223
diff
changeset
|
511 set ustatus [::http::status $utoken] |
|
aaf433ab696a
urllog: Improve error messages a bit.
Matti Hamalainen <ccr@tnsp.org>
parents:
223
diff
changeset
|
512 set uscode [::http::code $utoken] |
| 28 | 513 set ucode [::http::ncode $utoken] |
|
116
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
514 set udata [::http::data $utoken] |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
515 array set umeta [::http::meta $utoken] |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
516 ::http::cleanup $utoken |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
517 |
| 28 | 518 if {$ucode >= 200 && $ucode <= 309} { |
|
116
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
519 set uenc_doc "" |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
520 set uenc_http "" |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
521 set uencoding "" |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
522 |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
523 # Get information about specified character encodings |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
524 if {[info exists umeta(Content-Type)] && [regexp -nocase {charset\s*=\s*([a-z0-9._-]+)} $umeta(Content-Type) umatches uenc_http]} { |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
525 # Found character set encoding information in HTTP headers |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
526 } |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
527 |
|
150
52350ed97775
urllog: Cleanups, rename/move some global variables.
Matti Hamalainen <ccr@tnsp.org>
parents:
136
diff
changeset
|
528 if {[regexp -nocase -- "<meta.\*\?content=\"text/html.\*\?charset=(\[^\"\]*)\".\*\?/\?>" $udata umatches uenc_doc]} { |
|
116
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
529 # Found old style HTML meta tag with character set information |
|
150
52350ed97775
urllog: Cleanups, rename/move some global variables.
Matti Hamalainen <ccr@tnsp.org>
parents:
136
diff
changeset
|
530 } elseif {[regexp -nocase -- "<meta.\*\?charset=\"(\[^\"\]*)\".\*\?/\?>" $udata umatches uenc_doc]} { |
|
116
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
531 # Found HTML5 style meta tag with character set information |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
532 } |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
533 |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
534 # Make sanitized versions of the encoding strings |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
535 set uenc_http2 [urllog_sanitize_encoding $uenc_http] |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
536 set uenc_doc2 [urllog_sanitize_encoding $uenc_doc] |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
537 |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
538 # KLUDGE! |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
539 set uencoding $uenc_http2 |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
540 |
|
210
52cadf5a12b6
urllog: Disable some debug logging.
Matti Hamalainen <ccr@tnsp.org>
parents:
209
diff
changeset
|
541 # putlog "got charsets : http='$uenc_http', doc='$uenc_doc' / sanitized http='$uenc_http2', doc='$uenc_doc2'" |
|
150
52350ed97775
urllog: Cleanups, rename/move some global variables.
Matti Hamalainen <ccr@tnsp.org>
parents:
136
diff
changeset
|
542 |
|
52350ed97775
urllog: Cleanups, rename/move some global variables.
Matti Hamalainen <ccr@tnsp.org>
parents:
136
diff
changeset
|
543 |
|
116
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
544 # Check if the document has specified encoding |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
545 if {$uenc_doc != ""} { |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
546 # Does it differ from what HTTP says? |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
547 if {$uenc_http != "" && $uenc_doc != $uenc_http && $uenc_doc2 != $uenc_http2} { |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
548 # Yes, we will try reconverting |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
549 set uencoding $uenc_doc2 |
| 28 | 550 } |
|
116
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
551 } elseif {$uenc_http == ""} { |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
552 # If _NO_ known encoding of any kind, assume the default of iso8859-1 |
|
86
4c2b6482c08c
urllog: Different strategy for charset encoding conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
84
diff
changeset
|
553 set uencoding "iso8859-1" |
|
4c2b6482c08c
urllog: Different strategy for charset encoding conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
84
diff
changeset
|
554 } |
| 0 | 555 |
|
116
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
556 # Get the document title, if any |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
557 set urlTitle "" |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
558 if {[regexp -nocase -- "<title>(.\*\?)</title>" $udata umatches urlTitle]} { |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
559 # If character set conversion is required, do it now |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
560 if {$uencoding != ""} { |
|
210
52cadf5a12b6
urllog: Disable some debug logging.
Matti Hamalainen <ccr@tnsp.org>
parents:
209
diff
changeset
|
561 # putlog "conversion requested from $uencoding" |
|
116
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
562 if {[catch {set urlTitle [encoding convertfrom $uencoding $urlTitle]} cerrmsg]} { |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
563 urllog_log "Error in charset conversion: $cerrmsg" |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
564 } |
| 28 | 565 } |
|
150
52350ed97775
urllog: Cleanups, rename/move some global variables.
Matti Hamalainen <ccr@tnsp.org>
parents:
136
diff
changeset
|
566 |
|
52350ed97775
urllog: Cleanups, rename/move some global variables.
Matti Hamalainen <ccr@tnsp.org>
parents:
136
diff
changeset
|
567 # putlog "xxx: $uencoding : '$urlTitle'" |
|
52350ed97775
urllog: Cleanups, rename/move some global variables.
Matti Hamalainen <ccr@tnsp.org>
parents:
136
diff
changeset
|
568 # return 0 |
|
116
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
569 |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
570 # Convert some HTML entities to plaintext and do some cleanup |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
571 set utmp [urllog_convert_ent $urlTitle] |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
572 regsub -all "\r|\n|\t" $utmp " " utmp |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
573 regsub -all " *" $utmp " " utmp |
|
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
574 set urlTitle [string trim $utmp] |
| 28 | 575 } |
|
3
8003090caa35
Lots of code cleanups, add "fixer" for RasiaTube links (which suck) to point directly to Youtube.
Matti Hamalainen <ccr@tnsp.org>
parents:
0
diff
changeset
|
576 |
| 28 | 577 # Rasiatube hack |
| 578 if {[string match "*/rasiatube/view*" $urlStr]} { | |
| 579 set rasia 0 | |
|
118
e5f2961a6145
urllog: Improve rasiatube URL de-mangling.
Matti Hamalainen <ccr@tnsp.org>
parents:
117
diff
changeset
|
580 if {[regexp -nocase -- "<link rel=\"video_src\"\.\*\?file=(http://\[^&\]+)&" $udata umatches utmp]} { |
|
e5f2961a6145
urllog: Improve rasiatube URL de-mangling.
Matti Hamalainen <ccr@tnsp.org>
parents:
117
diff
changeset
|
581 regsub -all "\/v\/" $utmp "\/watch\?v=" urlStr |
| 28 | 582 set rasia 1 |
| 583 } else { | |
|
118
e5f2961a6145
urllog: Improve rasiatube URL de-mangling.
Matti Hamalainen <ccr@tnsp.org>
parents:
117
diff
changeset
|
584 if {[regexp -nocase -- "SWFObject.\"(\[^\"\]+)\", *\"flashvideo" $udata umatches utmp]} { |
|
e5f2961a6145
urllog: Improve rasiatube URL de-mangling.
Matti Hamalainen <ccr@tnsp.org>
parents:
117
diff
changeset
|
585 regsub "http:\/\/www.dailymotion.com\/swf\/" $utmp "http:\/\/www.dailymotion.com\/video\/" urlStr |
| 28 | 586 set rasia 1 |
| 587 } | |
| 588 } | |
| 589 if {$rasia != 0} { | |
| 590 urllog_log "RasiaTube mangler: $urlStr" | |
| 591 urllog_verb_msg $urlNick $urlChan "Korjataan haiseva rasiatube-linkki: $urlStr" | |
| 592 } | |
| 593 } | |
|
3
8003090caa35
Lots of code cleanups, add "fixer" for RasiaTube links (which suck) to point directly to Youtube.
Matti Hamalainen <ccr@tnsp.org>
parents:
0
diff
changeset
|
594 |
|
83
f171a9fb7b7b
urllog: Split urllog_add function to urllog_exists for checking whether given URL already exists in the database. Use urllog_exists where appropriate.
Matti Hamalainen <ccr@tnsp.org>
parents:
82
diff
changeset
|
595 # Check if the URL already exists, just in case we had some redirects |
|
f171a9fb7b7b
urllog: Split urllog_add function to urllog_exists for checking whether given URL already exists in the database. Use urllog_exists where appropriate.
Matti Hamalainen <ccr@tnsp.org>
parents:
82
diff
changeset
|
596 if {[urllog_exists $urlStr $urlNick $urlHost $urlChan]} { |
|
f171a9fb7b7b
urllog: Split urllog_add function to urllog_exists for checking whether given URL already exists in the database. Use urllog_exists where appropriate.
Matti Hamalainen <ccr@tnsp.org>
parents:
82
diff
changeset
|
597 urllog_addurl $urlStr $urlNick $urlHost $urlChan $urlTitle |
|
f171a9fb7b7b
urllog: Split urllog_add function to urllog_exists for checking whether given URL already exists in the database. Use urllog_exists where appropriate.
Matti Hamalainen <ccr@tnsp.org>
parents:
82
diff
changeset
|
598 } |
| 28 | 599 return 1 |
| 600 } else { | |
|
116
4f3edcf72987
urllog: Improvements in document / HTTP encoding handling and conversion.
Matti Hamalainen <ccr@tnsp.org>
parents:
115
diff
changeset
|
601 urllog_verb_msg $urlNick $urlChan "$urlmsg_errorgettingdoc ($ucode)" |
|
224
aaf433ab696a
urllog: Improve error messages a bit.
Matti Hamalainen <ccr@tnsp.org>
parents:
223
diff
changeset
|
602 urllog_log "Error fetching document: status=$ustatus, code=$ucode, scode=$uscode, url=$urlStr" |
| 28 | 603 } |
| 0 | 604 } |
| 605 | |
| 606 | |
| 607 #------------------------------------------------------------------------- | |
|
219
4e09bcc48851
urllog: Add settings for specifying channels where URL logging is active, and where !urlfind functionality works (separately, if so desired.)
Matti Hamalainen <ccr@tnsp.org>
parents:
218
diff
changeset
|
608 |
|
4e09bcc48851
urllog: Add settings for specifying channels where URL logging is active, and where !urlfind functionality works (separately, if so desired.)
Matti Hamalainen <ccr@tnsp.org>
parents:
218
diff
changeset
|
609 |
| 87 | 610 proc urllog_checkmsg {unick uhost uhand uchan utext} { |
|
219
4e09bcc48851
urllog: Add settings for specifying channels where URL logging is active, and where !urlfind functionality works (separately, if so desired.)
Matti Hamalainen <ccr@tnsp.org>
parents:
218
diff
changeset
|
611 global urllog_log_channels |
|
4e09bcc48851
urllog: Add settings for specifying channels where URL logging is active, and where !urlfind functionality works (separately, if so desired.)
Matti Hamalainen <ccr@tnsp.org>
parents:
218
diff
changeset
|
612 |
| 28 | 613 ### Check the nick |
| 87 | 614 if {$unick == "*"} { |
| 28 | 615 urllog_log "urllog_checkmsg: nick was wc, this should not happen." |
| 616 return 0 | |
| 617 } | |
| 0 | 618 |
|
219
4e09bcc48851
urllog: Add settings for specifying channels where URL logging is active, and where !urlfind functionality works (separately, if so desired.)
Matti Hamalainen <ccr@tnsp.org>
parents:
218
diff
changeset
|
619 ### Check the channel |
|
4e09bcc48851
urllog: Add settings for specifying channels where URL logging is active, and where !urlfind functionality works (separately, if so desired.)
Matti Hamalainen <ccr@tnsp.org>
parents:
218
diff
changeset
|
620 foreach akey [split $urllog_log_channels] { |
|
4e09bcc48851
urllog: Add settings for specifying channels where URL logging is active, and where !urlfind functionality works (separately, if so desired.)
Matti Hamalainen <ccr@tnsp.org>
parents:
218
diff
changeset
|
621 if {[string match $akey $uchan]} { |
|
4e09bcc48851
urllog: Add settings for specifying channels where URL logging is active, and where !urlfind functionality works (separately, if so desired.)
Matti Hamalainen <ccr@tnsp.org>
parents:
218
diff
changeset
|
622 ### Do the URL checking |
|
4e09bcc48851
urllog: Add settings for specifying channels where URL logging is active, and where !urlfind functionality works (separately, if so desired.)
Matti Hamalainen <ccr@tnsp.org>
parents:
218
diff
changeset
|
623 foreach str [split $utext " "] { |
|
221
b8bf9d7666b6
urllog: Improve URL / link matching.
Matti Hamalainen <ccr@tnsp.org>
parents:
219
diff
changeset
|
624 if {[regexp "((ftp|http|https)://\[^\[:space:\]\]+|^(www|ftp)\.\[^\[:space:\]\]+)" $str ulink]} { |
|
219
4e09bcc48851
urllog: Add settings for specifying channels where URL logging is active, and where !urlfind functionality works (separately, if so desired.)
Matti Hamalainen <ccr@tnsp.org>
parents:
218
diff
changeset
|
625 urllog_checkurl $str $unick $uhost $uchan |
|
4e09bcc48851
urllog: Add settings for specifying channels where URL logging is active, and where !urlfind functionality works (separately, if so desired.)
Matti Hamalainen <ccr@tnsp.org>
parents:
218
diff
changeset
|
626 } |
|
4e09bcc48851
urllog: Add settings for specifying channels where URL logging is active, and where !urlfind functionality works (separately, if so desired.)
Matti Hamalainen <ccr@tnsp.org>
parents:
218
diff
changeset
|
627 } |
|
4e09bcc48851
urllog: Add settings for specifying channels where URL logging is active, and where !urlfind functionality works (separately, if so desired.)
Matti Hamalainen <ccr@tnsp.org>
parents:
218
diff
changeset
|
628 return 0 |
| 28 | 629 } |
| 630 } | |
| 0 | 631 |
| 28 | 632 return 0 |
| 0 | 633 } |
| 634 | |
| 635 | |
| 636 #------------------------------------------------------------------------- | |
| 637 ### Parse arguments, find and show the results | |
| 638 proc urllog_find {unick uhand uchan utext upublic} { | |
|
62
6428b1bcb34b
urllog: Remove some global variable references where they are not used.
Matti Hamalainen <ccr@tnsp.org>
parents:
50
diff
changeset
|
639 global urllog_shorturl urldb |
| 28 | 640 global urllog_showmax_pub urllog_showmax_priv urlmsg_nomatch |
| 0 | 641 |
| 28 | 642 if {$upublic == 0} { |
| 643 set ulimit 5 | |
| 644 } else { | |
| 645 set ulimit 3 | |
| 646 } | |
|
19
9cf22053e5da
Repair !urlfind functionality.
Matti Hamalainen <ccr@tnsp.org>
parents:
18
diff
changeset
|
647 |
| 28 | 648 ### Parse the given command |
| 649 urllog_log "$unick/$uhand searched URL: $utext" | |
| 0 | 650 |
| 28 | 651 set ftokens [split $utext " "] |
| 652 set fpatlist "" | |
| 653 foreach ftoken $ftokens { | |
| 654 set fprefix [string range $ftoken 0 0] | |
| 655 set fpattern [string range $ftoken 1 end] | |
|
128
0d21b9d1d2b9
urllog: Improve search functionality.
Matti Hamalainen <ccr@tnsp.org>
parents:
127
diff
changeset
|
656 set qpattern "'%[urllog_escape $fpattern]%'" |
| 0 | 657 |
| 28 | 658 if {$fprefix == "-"} { |
|
128
0d21b9d1d2b9
urllog: Improve search functionality.
Matti Hamalainen <ccr@tnsp.org>
parents:
127
diff
changeset
|
659 lappend fpatlist "(url NOT LIKE $qpattern OR title NOT LIKE $qpattern)" |
| 28 | 660 } elseif {$fprefix == "%"} { |
|
128
0d21b9d1d2b9
urllog: Improve search functionality.
Matti Hamalainen <ccr@tnsp.org>
parents:
127
diff
changeset
|
661 lappend fpatlist "user LIKE $qpattern" |
| 28 | 662 } elseif {$fprefix == "@"} { |
| 663 # foo | |
|
112
fae3dd7a8b20
urllog: Oops, a typo in variable name. Fixed.
Matti Hamalainen <ccr@tnsp.org>
parents:
111
diff
changeset
|
664 } elseif {$fprefix == "+"} { |
|
128
0d21b9d1d2b9
urllog: Improve search functionality.
Matti Hamalainen <ccr@tnsp.org>
parents:
127
diff
changeset
|
665 lappend fpatlist "(url LIKE $qpattern OR title LIKE $qpattern)" |
| 28 | 666 } else { |
|
128
0d21b9d1d2b9
urllog: Improve search functionality.
Matti Hamalainen <ccr@tnsp.org>
parents:
127
diff
changeset
|
667 set qpattern "'%[urllog_escape $ftoken]%'" |
|
0d21b9d1d2b9
urllog: Improve search functionality.
Matti Hamalainen <ccr@tnsp.org>
parents:
127
diff
changeset
|
668 lappend fpatlist "(url LIKE $qpattern OR title LIKE $qpattern)" |
| 28 | 669 } |
| 670 } | |
|
19
9cf22053e5da
Repair !urlfind functionality.
Matti Hamalainen <ccr@tnsp.org>
parents:
18
diff
changeset
|
671 |
|
27
6e381916b016
Some fixes in the query mechanisms of QuoteDB and URLLog.
Matti Hamalainen <ccr@tnsp.org>
parents:
20
diff
changeset
|
672 if {[llength $fpatlist] > 0} { |
|
6e381916b016
Some fixes in the query mechanisms of QuoteDB and URLLog.
Matti Hamalainen <ccr@tnsp.org>
parents:
20
diff
changeset
|
673 set fquery "WHERE [join $fpatlist " AND "]" |
|
6e381916b016
Some fixes in the query mechanisms of QuoteDB and URLLog.
Matti Hamalainen <ccr@tnsp.org>
parents:
20
diff
changeset
|
674 } else { |
|
6e381916b016
Some fixes in the query mechanisms of QuoteDB and URLLog.
Matti Hamalainen <ccr@tnsp.org>
parents:
20
diff
changeset
|
675 set fquery "" |
|
6e381916b016
Some fixes in the query mechanisms of QuoteDB and URLLog.
Matti Hamalainen <ccr@tnsp.org>
parents:
20
diff
changeset
|
676 } |
| 68 | 677 |
| 28 | 678 set iresults 0 |
|
82
1bbc79f41a1c
urllog: Rename few variables for clarity.
Matti Hamalainen <ccr@tnsp.org>
parents:
81
diff
changeset
|
679 set usql "SELECT id AS uid, utime AS utime, url AS uurl, user AS uuser, host AS uhost FROM urls $fquery ORDER BY utime DESC LIMIT $ulimit" |
| 68 | 680 urldb eval $usql { |
| 28 | 681 incr iresults |
| 682 set shortURL $uurl | |
|
82
1bbc79f41a1c
urllog: Rename few variables for clarity.
Matti Hamalainen <ccr@tnsp.org>
parents:
81
diff
changeset
|
683 if {$urllog_shorturl != 0 && $uid != ""} { |
|
1bbc79f41a1c
urllog: Rename few variables for clarity.
Matti Hamalainen <ccr@tnsp.org>
parents:
81
diff
changeset
|
684 set shortURL "$shortURL [urllog_get_short $uid]" |
| 28 | 685 } |
| 686 urllog_msg $upublic $unick $uchan "#$iresults: $shortURL ($uuser@[urllog_ctime $utime])" | |
| 687 } | |
| 688 | |
| 689 if {$iresults == 0} { | |
| 690 # If no URLs were found | |
| 691 urllog_msg $upublic $unick $uchan $urlmsg_nomatch | |
| 692 } | |
| 0 | 693 |
| 28 | 694 return 0 |
| 0 | 695 } |
| 696 | |
| 697 | |
| 698 #------------------------------------------------------------------------- | |
| 699 ### Finding binded functions | |
| 700 proc urllog_pub_urlfind {unick uhost uhand uchan utext} { | |
|
219
4e09bcc48851
urllog: Add settings for specifying channels where URL logging is active, and where !urlfind functionality works (separately, if so desired.)
Matti Hamalainen <ccr@tnsp.org>
parents:
218
diff
changeset
|
701 global urllog_search_channels |
|
4e09bcc48851
urllog: Add settings for specifying channels where URL logging is active, and where !urlfind functionality works (separately, if so desired.)
Matti Hamalainen <ccr@tnsp.org>
parents:
218
diff
changeset
|
702 |
|
4e09bcc48851
urllog: Add settings for specifying channels where URL logging is active, and where !urlfind functionality works (separately, if so desired.)
Matti Hamalainen <ccr@tnsp.org>
parents:
218
diff
changeset
|
703 foreach akey [split $urllog_search_channels ";"] { |
|
4e09bcc48851
urllog: Add settings for specifying channels where URL logging is active, and where !urlfind functionality works (separately, if so desired.)
Matti Hamalainen <ccr@tnsp.org>
parents:
218
diff
changeset
|
704 if {[string match $akey $uchan]} { |
|
4e09bcc48851
urllog: Add settings for specifying channels where URL logging is active, and where !urlfind functionality works (separately, if so desired.)
Matti Hamalainen <ccr@tnsp.org>
parents:
218
diff
changeset
|
705 return [urllog_find $unick $uhand $uchan $utext 1] |
|
4e09bcc48851
urllog: Add settings for specifying channels where URL logging is active, and where !urlfind functionality works (separately, if so desired.)
Matti Hamalainen <ccr@tnsp.org>
parents:
218
diff
changeset
|
706 } |
|
4e09bcc48851
urllog: Add settings for specifying channels where URL logging is active, and where !urlfind functionality works (separately, if so desired.)
Matti Hamalainen <ccr@tnsp.org>
parents:
218
diff
changeset
|
707 } |
| 28 | 708 return 0 |
| 0 | 709 } |
| 710 | |
| 711 | |
| 712 proc urllog_msg_urlfind {unick uhost uhand utext} { | |
| 28 | 713 urllog_find $unick $uhand "" $utext 0 |
| 714 return 0 | |
|
3
8003090caa35
Lots of code cleanups, add "fixer" for RasiaTube links (which suck) to point directly to Youtube.
Matti Hamalainen <ccr@tnsp.org>
parents:
0
diff
changeset
|
715 } |
| 0 | 716 |
| 717 # end of script |