Mercurial > hg > lukkari
annotate update.sh @ 142:0e8b8fac920d
Check request instead of $_GET.
author | Matti Hamalainen <ccr@tnsp.org> |
---|---|
date | Sun, 31 Aug 2014 01:10:09 +0300 |
parents | 6f7a2f9dcad4 |
children | ca012374190c |
rev | line source |
---|---|
6
7fca87c41e17
Added data fetching and updating shellscript.
Matti Hamalainen <ccr@tnsp.org>
parents:
diff
changeset
|
1 #!/bin/sh |
7fca87c41e17
Added data fetching and updating shellscript.
Matti Hamalainen <ccr@tnsp.org>
parents:
diff
changeset
|
2 URLSUFFIX=".htm" |
7fca87c41e17
Added data fetching and updating shellscript.
Matti Hamalainen <ccr@tnsp.org>
parents:
diff
changeset
|
3 |
76
d1b65d9903ab
Clean up the update script slightly.
Matti Hamalainen <ccr@tnsp.org>
parents:
75
diff
changeset
|
4 function createdir() |
d1b65d9903ab
Clean up the update script slightly.
Matti Hamalainen <ccr@tnsp.org>
parents:
75
diff
changeset
|
5 { |
125 | 6 test ! -d "$1" && mkdir -p "$1" && chmod 751 "$1" |
76
d1b65d9903ab
Clean up the update script slightly.
Matti Hamalainen <ccr@tnsp.org>
parents:
75
diff
changeset
|
7 } |
d1b65d9903ab
Clean up the update script slightly.
Matti Hamalainen <ccr@tnsp.org>
parents:
75
diff
changeset
|
8 |
125 | 9 |
101
891bd3c93f96
Modularize update script and fetch list of classes data separately from the
Matti Hamalainen <ccr@tnsp.org>
parents:
97
diff
changeset
|
10 function fetch() |
891bd3c93f96
Modularize update script and fetch list of classes data separately from the
Matti Hamalainen <ccr@tnsp.org>
parents:
97
diff
changeset
|
11 { |
124
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
12 URLPREFIX="$1" |
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
13 CLASSFILE="$2" |
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
14 LISTFILE="$2.tmp" |
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
15 PATPREFIX="$3" |
101
891bd3c93f96
Modularize update script and fetch list of classes data separately from the
Matti Hamalainen <ccr@tnsp.org>
parents:
97
diff
changeset
|
16 |
124
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
17 if wget -q -O "$LISTFILE" "$URLPREFIX"; then |
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
18 perl -ne "if (/<a href=\"${PATPREFIX}([A-Z]{3}\d\S+)${URLSUFFIX}\">/) { print \"\$1\n\"; }" < "$LISTFILE" > "$CLASSFILE" |
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
19 echo -n "* Fetched classfile $CLASSFILE: " |
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
20 cat "$CLASSFILE" | wc -l |
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
21 fi |
101
891bd3c93f96
Modularize update script and fetch list of classes data separately from the
Matti Hamalainen <ccr@tnsp.org>
parents:
97
diff
changeset
|
22 } |
891bd3c93f96
Modularize update script and fetch list of classes data separately from the
Matti Hamalainen <ccr@tnsp.org>
parents:
97
diff
changeset
|
23 |
891bd3c93f96
Modularize update script and fetch list of classes data separately from the
Matti Hamalainen <ccr@tnsp.org>
parents:
97
diff
changeset
|
24 |
24
1b8260151e99
Get updates / data from upcoming work-in-progress timetables for next period also.
Matti Hamalainen <ccr@tnsp.org>
parents:
22
diff
changeset
|
25 function parse() |
1b8260151e99
Get updates / data from upcoming work-in-progress timetables for next period also.
Matti Hamalainen <ccr@tnsp.org>
parents:
22
diff
changeset
|
26 { |
124
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
27 # Create cache directories, if they do not exist |
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
28 OLDCACHEDIR="${CACHEDIR}old/" |
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
29 createdir "${OLDCACHEDIR}" |
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
30 createdir "${CACHEDIR}" |
46
9b5fa0f3812b
Improvements in the update logic.
Matti Hamalainen <ccr@tnsp.org>
parents:
42
diff
changeset
|
31 |
124
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
32 URLPREFIX="$1" |
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
33 CLASSFILE="$2" |
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
34 PATPREFIX="$3" |
6
7fca87c41e17
Added data fetching and updating shellscript.
Matti Hamalainen <ccr@tnsp.org>
parents:
diff
changeset
|
35 |
124
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
36 echo "<? \$origBaseURI = \"${URLPREFIX}${PATPREFIX}\"; \$origBaseExt = \".htm\"; ?>" > "${CACHEDIR}baseuri.data"; |
82
c553ad61e9c2
Make original data links work for both current and next period mode.
Matti Hamalainen <ccr@tnsp.org>
parents:
76
diff
changeset
|
37 |
124
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
38 test -e "$CLASSFILE" || return 0 |
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
39 |
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
40 echo "* Processing $CLASSFILE ..." |
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
41 cat "$CLASSFILE" | while read i; do |
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
42 parse=no |
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
43 INFILE="${CACHEDIR}${i}.html" |
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
44 ONFILE="${OLDCACHEDIR}${i}.html" |
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
45 DATAFILE="${i}.data" |
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
46 wget -q -O "${INFILE}.new" "${URLPREFIX}${PATPREFIX}${i}${URLSUFFIX}" |
6
7fca87c41e17
Added data fetching and updating shellscript.
Matti Hamalainen <ccr@tnsp.org>
parents:
diff
changeset
|
47 |
124
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
48 if test -e "${INFILE}.new"; then |
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
49 # New data fetched, does old file exist? |
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
50 if test -e "$INFILE"; then |
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
51 # Yes, do a diff |
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
52 if ! diff -u "$INFILE" "$INFILE.new" > "$INFILE.diff"; then |
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
53 # There were differences, do a parse |
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
54 parse=yes |
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
55 mv "$INFILE" "$ONFILE" && \ |
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
56 mv "$INFILE.new" "$INFILE" |
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
57 else |
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
58 # No changes, apparently .. remove the new one |
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
59 rm -f "$INFILE.new" "$INFILE.diff" |
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
60 fi |
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
61 else |
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
62 # No old file, parse new data |
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
63 mv "$INFILE.new" "$INFILE" |
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
64 parse=yes |
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
65 fi |
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
66 fi |
6
7fca87c41e17
Added data fetching and updating shellscript.
Matti Hamalainen <ccr@tnsp.org>
parents:
diff
changeset
|
67 |
124
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
68 # No new file fetched, does datafile exist? |
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
69 if test ! -e "${CACHEDIR}${DATAFILE}"; then |
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
70 # No, try to parse it if old file input exists |
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
71 parse=yes |
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
72 fi |
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
73 |
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
74 # Parsing of old data requested? |
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
75 if test "x$parse" = "xyes" -a -e "$INFILE"; then |
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
76 if test -e "${CACHEDIR}${DATAFILE}"; then |
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
77 mv "${CACHEDIR}${DATAFILE}" "${OLDCACHEDIR}${DATAFILE}" |
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
78 fi |
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
79 echo "Parsing $i" |
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
80 perl parsedata.pl -php "$INFILE" -o "${CACHEDIR}${DATAFILE}" |
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
81 #perl parsedata.pl -xml "$INFILE" -o "${CACHEDIR}${i}.xml" |
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
82 fi |
4547a239b0fd
Indentation cleanup, use spaces instead of tabs.
Matti Hamalainen <ccr@tnsp.org>
parents:
122
diff
changeset
|
83 done |
24
1b8260151e99
Get updates / data from upcoming work-in-progress timetables for next period also.
Matti Hamalainen <ccr@tnsp.org>
parents:
22
diff
changeset
|
84 } |
6
7fca87c41e17
Added data fetching and updating shellscript.
Matti Hamalainen <ccr@tnsp.org>
parents:
diff
changeset
|
85 |
76
d1b65d9903ab
Clean up the update script slightly.
Matti Hamalainen <ccr@tnsp.org>
parents:
75
diff
changeset
|
86 # Fix permissions |
d1b65d9903ab
Clean up the update script slightly.
Matti Hamalainen <ccr@tnsp.org>
parents:
75
diff
changeset
|
87 chmod 751 . && chmod 644 *.php *.css && chmod 600 coursecache.txt classes.txt classes_next.txt |
d1b65d9903ab
Clean up the update script slightly.
Matti Hamalainen <ccr@tnsp.org>
parents:
75
diff
changeset
|
88 |
35
4d9354abda73
Update fetching URLs and parameters to match the latest changes on OAMK's web.
Matti Hamalainen <ccr@tnsp.org>
parents:
34
diff
changeset
|
89 CACHEDIR="cache/" |
134
6f7a2f9dcad4
Fix some errors due to undefined variables in case the class data is not
Matti Hamalainen <ccr@tnsp.org>
parents:
125
diff
changeset
|
90 fetch "http://www.oamk.fi/~heikkim/riihi1/luokat/" "classes.txt" "Ryh._" |
6f7a2f9dcad4
Fix some errors due to undefined variables in case the class data is not
Matti Hamalainen <ccr@tnsp.org>
parents:
125
diff
changeset
|
91 parse "http://www.oamk.fi/~heikkim/riihi1/luokat/" "classes.txt" "Ryh._" |
6f7a2f9dcad4
Fix some errors due to undefined variables in case the class data is not
Matti Hamalainen <ccr@tnsp.org>
parents:
125
diff
changeset
|
92 #parse "http://www.oamk.fi/tyojarjestykset/otek/luokat/" "classes.txt" "OR_" |
29
ac51fc10414f
Add support for URL prefix pattern in updates.
Matti Hamalainen <ccr@tnsp.org>
parents:
24
diff
changeset
|
93 |
122 | 94 |
31
dbe7ff545293
Add support for fetching and showing data for next/upcoming period.
Matti Hamalainen <ccr@tnsp.org>
parents:
29
diff
changeset
|
95 CACHEDIR="cache-next/" |
134
6f7a2f9dcad4
Fix some errors due to undefined variables in case the class data is not
Matti Hamalainen <ccr@tnsp.org>
parents:
125
diff
changeset
|
96 #fetch "http://www.oamk.fi/~heikkim/riihi1/luokat/" "classes.txt" "Ryh._" |
6f7a2f9dcad4
Fix some errors due to undefined variables in case the class data is not
Matti Hamalainen <ccr@tnsp.org>
parents:
125
diff
changeset
|
97 #parse "http://www.oamk.fi/~heikkim/riihi1/luokat/" "classes.txt" "Ryh._" |
102
bf7a9f63bd82
Add some comments to the update script.
Matti Hamalainen <ccr@tnsp.org>
parents:
101
diff
changeset
|
98 |
bf7a9f63bd82
Add some comments to the update script.
Matti Hamalainen <ccr@tnsp.org>
parents:
101
diff
changeset
|
99 |
bf7a9f63bd82
Add some comments to the update script.
Matti Hamalainen <ccr@tnsp.org>
parents:
101
diff
changeset
|
100 # http://www.oamk.fi/~heikkim/riihi[1-5]/ |
bf7a9f63bd82
Add some comments to the update script.
Matti Hamalainen <ccr@tnsp.org>
parents:
101
diff
changeset
|
101 # http://www.oamk.fi/~heikkim/Luhti[1-5]/ |