Mercurial > hg > lukkari

annotate update.sh @ 122:8cbd07999b66

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Update to latest upcoming.
author Matti Hamalainen <ccr@tnsp.org>
date Tue, 11 Feb 2014 12:28:16 +0200
parents ee258438635f
children 4547a239b0fd
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
6
7fca87c41e17 Added data fetching and updating shellscript.
Matti Hamalainen <ccr@tnsp.org>
parents:
diff changeset
1 #!/bin/sh
7fca87c41e17 Added data fetching and updating shellscript.
Matti Hamalainen <ccr@tnsp.org>
parents:
diff changeset
2 URLSUFFIX=".htm"
7fca87c41e17 Added data fetching and updating shellscript.
Matti Hamalainen <ccr@tnsp.org>
parents:
diff changeset
3
76
d1b65d9903ab Clean up the update script slightly.
Matti Hamalainen <ccr@tnsp.org>
parents: 75
diff changeset
4 function createdir()
d1b65d9903ab Clean up the update script slightly.
Matti Hamalainen <ccr@tnsp.org>
parents: 75
diff changeset
5 {
d1b65d9903ab Clean up the update script slightly.
Matti Hamalainen <ccr@tnsp.org>
parents: 75
diff changeset
6 if test ! -d "$1"; then
d1b65d9903ab Clean up the update script slightly.
Matti Hamalainen <ccr@tnsp.org>
parents: 75
diff changeset
7 mkdir -p "$1" && chmod 751 "$1"
d1b65d9903ab Clean up the update script slightly.
Matti Hamalainen <ccr@tnsp.org>
parents: 75
diff changeset
8 fi
d1b65d9903ab Clean up the update script slightly.
Matti Hamalainen <ccr@tnsp.org>
parents: 75
diff changeset
9 }
d1b65d9903ab Clean up the update script slightly.
Matti Hamalainen <ccr@tnsp.org>
parents: 75
diff changeset
10
101
891bd3c93f96 Modularize update script and fetch list of classes data separately from the
Matti Hamalainen <ccr@tnsp.org>
parents: 97
diff changeset
11 function fetch()
891bd3c93f96 Modularize update script and fetch list of classes data separately from the
Matti Hamalainen <ccr@tnsp.org>
parents: 97
diff changeset
12 {
891bd3c93f96 Modularize update script and fetch list of classes data separately from the
Matti Hamalainen <ccr@tnsp.org>
parents: 97
diff changeset
13 URLPREFIX="$1"
891bd3c93f96 Modularize update script and fetch list of classes data separately from the
Matti Hamalainen <ccr@tnsp.org>
parents: 97
diff changeset
14 CLASSFILE="$2"
891bd3c93f96 Modularize update script and fetch list of classes data separately from the
Matti Hamalainen <ccr@tnsp.org>
parents: 97
diff changeset
15 LISTFILE="$2.tmp"
891bd3c93f96 Modularize update script and fetch list of classes data separately from the
Matti Hamalainen <ccr@tnsp.org>
parents: 97
diff changeset
16 PATPREFIX="$3"
891bd3c93f96 Modularize update script and fetch list of classes data separately from the
Matti Hamalainen <ccr@tnsp.org>
parents: 97
diff changeset
17
891bd3c93f96 Modularize update script and fetch list of classes data separately from the
Matti Hamalainen <ccr@tnsp.org>
parents: 97
diff changeset
18 if wget -q -O "$LISTFILE" "$URLPREFIX"; then
891bd3c93f96 Modularize update script and fetch list of classes data separately from the
Matti Hamalainen <ccr@tnsp.org>
parents: 97
diff changeset
19 perl -ne "if (/<a href=\"${PATPREFIX}([A-Z]{3}\d\S+)${URLSUFFIX}\">/) { print \"\$1\n\"; }" < "$LISTFILE" > "$CLASSFILE"
891bd3c93f96 Modularize update script and fetch list of classes data separately from the
Matti Hamalainen <ccr@tnsp.org>
parents: 97
diff changeset
20 echo -n "* Fetched classfile $CLASSFILE: "
891bd3c93f96 Modularize update script and fetch list of classes data separately from the
Matti Hamalainen <ccr@tnsp.org>
parents: 97
diff changeset
21 cat "$CLASSFILE" | wc -l
891bd3c93f96 Modularize update script and fetch list of classes data separately from the
Matti Hamalainen <ccr@tnsp.org>
parents: 97
diff changeset
22 fi
891bd3c93f96 Modularize update script and fetch list of classes data separately from the
Matti Hamalainen <ccr@tnsp.org>
parents: 97
diff changeset
23 }
891bd3c93f96 Modularize update script and fetch list of classes data separately from the
Matti Hamalainen <ccr@tnsp.org>
parents: 97
diff changeset
24
891bd3c93f96 Modularize update script and fetch list of classes data separately from the
Matti Hamalainen <ccr@tnsp.org>
parents: 97
diff changeset
25
24
1b8260151e99 Get updates / data from upcoming work-in-progress timetables for next period also.
Matti Hamalainen <ccr@tnsp.org>
parents: 22
diff changeset
26 function parse()
1b8260151e99 Get updates / data from upcoming work-in-progress timetables for next period also.
Matti Hamalainen <ccr@tnsp.org>
parents: 22
diff changeset
27 {
46
9b5fa0f3812b Improvements in the update logic.
Matti Hamalainen <ccr@tnsp.org>
parents: 42
diff changeset
28 # Create cache directories, if they do not exist
47
ceca06576918 Change backup (old) cache directories.
Matti Hamalainen <ccr@tnsp.org>
parents: 46
diff changeset
29 OLDCACHEDIR="${CACHEDIR}old/"
76
d1b65d9903ab Clean up the update script slightly.
Matti Hamalainen <ccr@tnsp.org>
parents: 75
diff changeset
30 createdir "${OLDCACHEDIR}"
d1b65d9903ab Clean up the update script slightly.
Matti Hamalainen <ccr@tnsp.org>
parents: 75
diff changeset
31 createdir "${CACHEDIR}"
46
9b5fa0f3812b Improvements in the update logic.
Matti Hamalainen <ccr@tnsp.org>
parents: 42
diff changeset
32
24
1b8260151e99 Get updates / data from upcoming work-in-progress timetables for next period also.
Matti Hamalainen <ccr@tnsp.org>
parents: 22
diff changeset
33 URLPREFIX="$1"
1b8260151e99 Get updates / data from upcoming work-in-progress timetables for next period also.
Matti Hamalainen <ccr@tnsp.org>
parents: 22
diff changeset
34 CLASSFILE="$2"
29
ac51fc10414f Add support for URL prefix pattern in updates.
Matti Hamalainen <ccr@tnsp.org>
parents: 24
diff changeset
35 PATPREFIX="$3"
6
7fca87c41e17 Added data fetching and updating shellscript.
Matti Hamalainen <ccr@tnsp.org>
parents:
diff changeset
36
82
c553ad61e9c2 Make original data links work for both current and next period mode.
Matti Hamalainen <ccr@tnsp.org>
parents: 76
diff changeset
37 echo "<? \$origBaseURI = \"${URLPREFIX}${PATPREFIX}\"; \$origBaseExt = \".htm\"; ?>" > "${CACHEDIR}baseuri.data";
c553ad61e9c2 Make original data links work for both current and next period mode.
Matti Hamalainen <ccr@tnsp.org>
parents: 76
diff changeset
38
24
1b8260151e99 Get updates / data from upcoming work-in-progress timetables for next period also.
Matti Hamalainen <ccr@tnsp.org>
parents: 22
diff changeset
39 if test -e "$CLASSFILE"; then
97
acb600abc0c7 Few minor improvements.
Matti Hamalainen <ccr@tnsp.org>
parents: 82
diff changeset
40 echo "* Processing $CLASSFILE ..."
6
7fca87c41e17 Added data fetching and updating shellscript.
Matti Hamalainen <ccr@tnsp.org>
parents:
diff changeset
41 cat "$CLASSFILE" | while read i; do
7fca87c41e17 Added data fetching and updating shellscript.
Matti Hamalainen <ccr@tnsp.org>
parents:
diff changeset
42 parse=no
7fca87c41e17 Added data fetching and updating shellscript.
Matti Hamalainen <ccr@tnsp.org>
parents:
diff changeset
43 INFILE="${CACHEDIR}${i}.html"
46
9b5fa0f3812b Improvements in the update logic.
Matti Hamalainen <ccr@tnsp.org>
parents: 42
diff changeset
44 ONFILE="${OLDCACHEDIR}${i}.html"
9b5fa0f3812b Improvements in the update logic.
Matti Hamalainen <ccr@tnsp.org>
parents: 42
diff changeset
45 DATAFILE="${i}.data"
9b5fa0f3812b Improvements in the update logic.
Matti Hamalainen <ccr@tnsp.org>
parents: 42
diff changeset
46 wget -q -O "${INFILE}.new" "${URLPREFIX}${PATPREFIX}${i}${URLSUFFIX}"
6
7fca87c41e17 Added data fetching and updating shellscript.
Matti Hamalainen <ccr@tnsp.org>
parents:
diff changeset
47
46
9b5fa0f3812b Improvements in the update logic.
Matti Hamalainen <ccr@tnsp.org>
parents: 42
diff changeset
48 if test -e "${INFILE}.new"; then
6
7fca87c41e17 Added data fetching and updating shellscript.
Matti Hamalainen <ccr@tnsp.org>
parents:
diff changeset
49 # New data fetched, does old file exist?
7fca87c41e17 Added data fetching and updating shellscript.
Matti Hamalainen <ccr@tnsp.org>
parents:
diff changeset
50 if test -e "$INFILE"; then
7fca87c41e17 Added data fetching and updating shellscript.
Matti Hamalainen <ccr@tnsp.org>
parents:
diff changeset
51 # Yes, do a diff
22
f025d36926a1 Make backup copies of old versions of data files when updating.
Matti Hamalainen <ccr@tnsp.org>
parents: 21
diff changeset
52 if ! diff -u "$INFILE" "$INFILE.new" > "$INFILE.diff"; then
6
7fca87c41e17 Added data fetching and updating shellscript.
Matti Hamalainen <ccr@tnsp.org>
parents:
diff changeset
53 # There were differences, do a parse
7fca87c41e17 Added data fetching and updating shellscript.
Matti Hamalainen <ccr@tnsp.org>
parents:
diff changeset
54 parse=yes
46
9b5fa0f3812b Improvements in the update logic.
Matti Hamalainen <ccr@tnsp.org>
parents: 42
diff changeset
55 mv "$INFILE" "$ONFILE" && \
6
7fca87c41e17 Added data fetching and updating shellscript.
Matti Hamalainen <ccr@tnsp.org>
parents:
diff changeset
56 mv "$INFILE.new" "$INFILE"
46
9b5fa0f3812b Improvements in the update logic.
Matti Hamalainen <ccr@tnsp.org>
parents: 42
diff changeset
57 else
9b5fa0f3812b Improvements in the update logic.
Matti Hamalainen <ccr@tnsp.org>
parents: 42
diff changeset
58 # No changes, apparently .. remove the new one
9b5fa0f3812b Improvements in the update logic.
Matti Hamalainen <ccr@tnsp.org>
parents: 42
diff changeset
59 rm -f "$INFILE.new" "$INFILE.diff"
6
7fca87c41e17 Added data fetching and updating shellscript.
Matti Hamalainen <ccr@tnsp.org>
parents:
diff changeset
60 fi
7fca87c41e17 Added data fetching and updating shellscript.
Matti Hamalainen <ccr@tnsp.org>
parents:
diff changeset
61 else
7fca87c41e17 Added data fetching and updating shellscript.
Matti Hamalainen <ccr@tnsp.org>
parents:
diff changeset
62 # No old file, parse new data
7fca87c41e17 Added data fetching and updating shellscript.
Matti Hamalainen <ccr@tnsp.org>
parents:
diff changeset
63 mv "$INFILE.new" "$INFILE"
7fca87c41e17 Added data fetching and updating shellscript.
Matti Hamalainen <ccr@tnsp.org>
parents:
diff changeset
64 parse=yes
7fca87c41e17 Added data fetching and updating shellscript.
Matti Hamalainen <ccr@tnsp.org>
parents:
diff changeset
65 fi
46
9b5fa0f3812b Improvements in the update logic.
Matti Hamalainen <ccr@tnsp.org>
parents: 42
diff changeset
66 fi
9b5fa0f3812b Improvements in the update logic.
Matti Hamalainen <ccr@tnsp.org>
parents: 42
diff changeset
67 # No new file fetched, does datafile exist?
9b5fa0f3812b Improvements in the update logic.
Matti Hamalainen <ccr@tnsp.org>
parents: 42
diff changeset
68 if test ! -e "${CACHEDIR}${DATAFILE}"; then
9b5fa0f3812b Improvements in the update logic.
Matti Hamalainen <ccr@tnsp.org>
parents: 42
diff changeset
69 # No, try to parse it if old file input exists
9b5fa0f3812b Improvements in the update logic.
Matti Hamalainen <ccr@tnsp.org>
parents: 42
diff changeset
70 parse=yes
6
7fca87c41e17 Added data fetching and updating shellscript.
Matti Hamalainen <ccr@tnsp.org>
parents:
diff changeset
71 fi
7fca87c41e17 Added data fetching and updating shellscript.
Matti Hamalainen <ccr@tnsp.org>
parents:
diff changeset
72
7fca87c41e17 Added data fetching and updating shellscript.
Matti Hamalainen <ccr@tnsp.org>
parents:
diff changeset
73 # Parsing of old data requested?
7fca87c41e17 Added data fetching and updating shellscript.
Matti Hamalainen <ccr@tnsp.org>
parents:
diff changeset
74 if test "x$parse" = "xyes" -a -e "$INFILE"; then
46
9b5fa0f3812b Improvements in the update logic.
Matti Hamalainen <ccr@tnsp.org>
parents: 42
diff changeset
75 if test -e "${CACHEDIR}${DATAFILE}"; then
9b5fa0f3812b Improvements in the update logic.
Matti Hamalainen <ccr@tnsp.org>
parents: 42
diff changeset
76 mv "${CACHEDIR}${DATAFILE}" "${OLDCACHEDIR}${DATAFILE}"
22
f025d36926a1 Make backup copies of old versions of data files when updating.
Matti Hamalainen <ccr@tnsp.org>
parents: 21
diff changeset
77 fi
34
1f6be8457912 Script output cosmetics.
Matti Hamalainen <ccr@tnsp.org>
parents: 31
diff changeset
78 echo "Parsing $i"
116
ee258438635f Disable generation of XML files, as they are not used really.
Matti Hamalainen <ccr@tnsp.org>
parents: 102
diff changeset
79 perl parsedata.pl -php "$INFILE" -o "${CACHEDIR}${DATAFILE}"
ee258438635f Disable generation of XML files, as they are not used really.
Matti Hamalainen <ccr@tnsp.org>
parents: 102
diff changeset
80 # perl parsedata.pl -xml "$INFILE" -o "${CACHEDIR}${i}.xml"
6
7fca87c41e17 Added data fetching and updating shellscript.
Matti Hamalainen <ccr@tnsp.org>
parents:
diff changeset
81 fi
7fca87c41e17 Added data fetching and updating shellscript.
Matti Hamalainen <ccr@tnsp.org>
parents:
diff changeset
82 done
24
1b8260151e99 Get updates / data from upcoming work-in-progress timetables for next period also.
Matti Hamalainen <ccr@tnsp.org>
parents: 22
diff changeset
83 fi
1b8260151e99 Get updates / data from upcoming work-in-progress timetables for next period also.
Matti Hamalainen <ccr@tnsp.org>
parents: 22
diff changeset
84 }
6
7fca87c41e17 Added data fetching and updating shellscript.
Matti Hamalainen <ccr@tnsp.org>
parents:
diff changeset
85
76
d1b65d9903ab Clean up the update script slightly.
Matti Hamalainen <ccr@tnsp.org>
parents: 75
diff changeset
86 # Fix permissions
d1b65d9903ab Clean up the update script slightly.
Matti Hamalainen <ccr@tnsp.org>
parents: 75
diff changeset
87 chmod 751 . && chmod 644 *.php *.css && chmod 600 coursecache.txt classes.txt classes_next.txt
d1b65d9903ab Clean up the update script slightly.
Matti Hamalainen <ccr@tnsp.org>
parents: 75
diff changeset
88
35
4d9354abda73 Update fetching URLs and parameters to match the latest changes on OAMK's web.
Matti Hamalainen <ccr@tnsp.org>
parents: 34
diff changeset
89 CACHEDIR="cache/"
101
891bd3c93f96 Modularize update script and fetch list of classes data separately from the
Matti Hamalainen <ccr@tnsp.org>
parents: 97
diff changeset
90 fetch "http://www.oamk.fi/tyojarjestykset/otek/luokat/" "classes.txt" "OR_"
122
8cbd07999b66 Update to latest upcoming.
Matti Hamalainen <ccr@tnsp.org>
parents: 116
diff changeset
91 fetch "http://www.oamk.fi/~heikkim/Luhti3/Ryhm%84t/" "classes.txt" "Ryh._"
65
2cfb0a7eac9b Rename some data files.
Matti Hamalainen <ccr@tnsp.org>
parents: 47
diff changeset
92 parse "http://www.oamk.fi/tyojarjestykset/otek/luokat/" "classes.txt" "OR_"
29
ac51fc10414f Add support for URL prefix pattern in updates.
Matti Hamalainen <ccr@tnsp.org>
parents: 24
diff changeset
93
122
8cbd07999b66 Update to latest upcoming.
Matti Hamalainen <ccr@tnsp.org>
parents: 116
diff changeset
94
31
dbe7ff545293 Add support for fetching and showing data for next/upcoming period.
Matti Hamalainen <ccr@tnsp.org>
parents: 29
diff changeset
95 CACHEDIR="cache-next/"
122
8cbd07999b66 Update to latest upcoming.
Matti Hamalainen <ccr@tnsp.org>
parents: 116
diff changeset
96 fetch "http://www.oamk.fi/~heikkim/Luhti4/Ryhm%84t/" "classes_next.txt" "Ryh._"
8cbd07999b66 Update to latest upcoming.
Matti Hamalainen <ccr@tnsp.org>
parents: 116
diff changeset
97 parse "http://www.oamk.fi/~heikkim/Luhti4/Ryhm%84t/" "classes_next.txt" "Ryh._"
102
bf7a9f63bd82 Add some comments to the update script.
Matti Hamalainen <ccr@tnsp.org>
parents: 101
diff changeset
98
bf7a9f63bd82 Add some comments to the update script.
Matti Hamalainen <ccr@tnsp.org>
parents: 101
diff changeset
99
bf7a9f63bd82 Add some comments to the update script.
Matti Hamalainen <ccr@tnsp.org>
parents: 101
diff changeset
100 # http://www.oamk.fi/~heikkim/riihi[1-5]/
bf7a9f63bd82 Add some comments to the update script.
Matti Hamalainen <ccr@tnsp.org>
parents: 101
diff changeset
101 # http://www.oamk.fi/~heikkim/Luhti[1-5]/